Data without context
23rd February 2006 by Alice Brink, ABC
I don’t know who said “data without context is trivia” (and I just Googled it and came up empty), but I think that phrase lies at the heart of the PR industry’s hesitation to invest in measurement. There was a minor firestorm recently ignited by a post in Houston PR consultant John Wagner’s blog saying that he thinks much of today’s measurement is meaningless and that “intuition” is an equally valid approach to assessing whether or not a PR campaign is successful. Katie Paine fired back, calling John a dinosaur and a “measurement menace,” and Shel Holtz got his licks in as well (earning “measurement maven” status from Katie).
The problem is that we sometimes view measurement in a vacuum as just a lot of statistics (oh, no, math…) that are an afterthought to a campaign. It becomes just a bunch of data without context, whether it’s ad values (boo, hiss!) or eyeballs reached or cost per impression. Those output numbers only take on meaning and move beyond trivia status when we put them in the context of objectives established at the outset that articulate what we are hoping to achieve — the classic question, “what do we want the audience to understand, believe or do as a result of our communications?” Then we can see how the visibility we generated contributed to the final outcomes, shaping knowledge, opinions and behavior. I once thought that it was only important to measure the final outcomes, but I have come to believe we need to measure both visibility and outcomes in order to demonstrate the value we contribute to the equation.
February 26th, 2006 at 11:32 am
I couldn’t agree, more, Alice! I also beleive that none of us pay anough attention to pre-measurement - whether focus groups, phone surveys or written questionnaires. Just think how much more effective our work would be IF we put it in context and if we did the up front stuff to make sure we’re on the right track. Objectives that link the business to the audience mindset - now there’s a novel concept!
February 27th, 2006 at 5:39 am
Alice:
Good to see you are blogging! Welcome.
Re: the subject at hand … my comments are important to put into context, as well. My point about intuition is based on “real world” situations where measurement is not possible, affordable or allowed.
Yes, in an ideal world we would pre- and post-measure every activity. I am guessing that very few organizations are interested in doing so. We may not like that, but it’s reality. Sure, push for measurement whenever possible, but at the end of the day, the client makes the call. Are we to walk away from client work that doesn’t involve associated focus groups, follow-up surveys and other activities? I’d like to meet the person who is doing so.
This argument reminds me of the “seat at the table” navel-gazing that has plagued communicators for so long. We are always chastising our profession and its participants for not doing or being something that is often out of our control.
In other words, no one is arguing that measurement is a bad thing, but I refuse to feel badly if I complete work that is not measured by an often meaningless number.
On a related note … when bloggers who sell measurement tools for a living tell you that some new metric is a must-have, I believe you have to take it with a grain of salt.
February 27th, 2006 at 2:21 pm
You are are right, John, you are guessing that few organizations are interested in pre- and post-measurement of every activity. However, your guesses do not include the many organizations (I would guess most) who view this type of measurement as a critical part of excellent communication management and actually budget for it when planning communication programs. Check out organizations who use a balanced scorecard approach for corporate management. Communication is right up there as a part of corporate strategy that demands measurement and is a signifanct part of the metrics of a balanced scroecard.
I have yet to meet a CFO who did not feel a budget (the pre) and cost reporting (the post-) should not be a part of every financial activity. Those CFOs and other senior executives that I deal with readily understand communication strategy and expect benchmarks and measureable targets. They also look for evalaution of performance from the communication managers.
This is not the ideal world, it is the real world where accountability for results is an expectation not an irritation.
If you are collecting meaningless numbers you are not measuring the right things or you have programs without measurable objectives. Perhaps this is why measurement is such an important criteria for judging Gold Quill entries.
Oh, by the way, I am a blogger who measures for a successful living. I have opinions that are worth more than a grain of salt. I do not sell tools and neither does Alice to the best of my knowledge.
February 27th, 2006 at 3:38 pm
Tudor:
I just don’t understand why the measurement gurus get so irritated by any suggestion that not every client or company is interested in measuring every activity.
You are a respected measurement consultant. Of course you work with companies who see value in measurement. They wouldn’t seek you out if they didn’t.
But there is indeed a whole other world out there — and it’s not just small companies — who don’t want to invest the resources necessary to truly measure results accurately. So they rely on meaningless data, reporting that isn’t of value or nothing at all. I’ve had large clients — big-name brand companies — who refused any kind of reporting other than ad value. Should I have fired the client — our firm’s largest? Of course not.
So it’s not something I’m making up. Those clients do exist out there. Why does that fact frustrate the consultants so much? To me, it’s a sign that there is lots of potential for your work … if consultants will stop criticizing and start educating.
And Tudor, I’ve worked for companies that used scorecard reporting so I’ve seen first-hand how they compile and measure internal and external communication results. I’m not arguing that it’s not of value. Of course it is. I’ve also won two Gold Quills and been a finalist several other times, so I’m no stranger to measurement.
My point about the selling of tools was not related to you or Alice, who I’ve known for probably 20 years. It was related to the discussion she referenced in her post.
February 28th, 2006 at 8:06 am
What an interesting conversation with good points raised to support all points of view. To Tudor’s point and speaking of Gold Quill - having judged literally thousands of entries at the coordinator and international levels of the past 12 years,(and won a few myself) I find it disconcerting that the weakest skill, bar none, is the ability to set meaning objectives aligned with the needs of the business and to measure results against those objectives.
I have also found that smart communicators with little or no budget for measurement do a darn fine job by looking at the big picture from an enterprise-wide perspective and using their creativity and business saavy to set meaningful objectives, measure and report results that do make a difference to the business.
I believe the failure to measure the impact of communication activities on business is the Achilles heel of our profession. It holds us back from being credible partners in the business and until the mass majority of us conquer this “Mount Everest” we will continue to be viewed as tacticians.
To your point, John - meaningful measurement is simply a matter of being dedicated to supporting the business objectives and strategically aligning communication to that end.
And if it is as you say - that large, brand name companies are not interested in anything more than ad value - then perhaps the challenge for consultants AND those of us who work in organizations - large or small - is one of education. Perhaps if we take seriously the task of shifting perception of the value of communication to the business and in doing so, to create a stronger reputation, greater trust, respect nad credibility for communicators everywhere. And that is one of the ways that all of us who understand the value of measurement can give back to the profession. It’s long past time to stop being the shoemaker’s kids.
March 1st, 2006 at 9:15 am
I just wanted to add one point on evaluation and context. Going back to what Alice said originally: “data without context is trivia”. Evaluation doesn’t have to focus on data alone. Look to other fields for inspiration. I’ve done evaluation of training programs and what makes an impact with clients are testimonies and success stories: “This is what I do differently as a result of the training”. Sure data drawn from measuring change of attitudes and behaviour counts but clients ultimately relate better to impact illustrated through stories.
March 1st, 2006 at 4:22 pm
Evaluation doesn’t have to focus on data alone.
Glenn, this is an excellent point and it’s related to what I was trying to say in my original post awhile back that got me labeled such a bad guy.
I called it intuition, which was in retrospect a poor choice of words. What I should have said is that a good communicator can find ways to measure results that don’t necessarily rely on expensive, off-the-shelf toolkits or wacky formulas. Your example is perfect.
In fact, in my opinion, good communicators are always measuring the impact of their work. They just don’t always do it in canned or obvious ways.
I’ve also seen many a weak communicator hide behind the need for research as either a way to make extra money or avoid ever really producing anything.
I’ve gotten a lot of gigs over the years from clients who hired the “right firm” but wound up with nothing more than big invoices and a binder full of useless data. They’ve called me in to actually accomplish something. In those scenarios, who’s more valuable to the client — the maven or the “menace”??
March 3rd, 2006 at 12:58 pm
As we communicators continue to scrap amongst ourselves over seats at tables and vendor management and what-all, out there in the real world there is serious work to be done.
The practice of communication/pr needs to change, and the will to change is what differentiates thriving organizations from those teetering on the edge of the dustbin.
One path to change is by embracing data as a concept. In fact, many companies have taken their communication teams to task at budget time because “communication is not quantitative” or “communicators can’t say what value they bring to the bottom line.”
Hence, as a profession, we are seeking to prove ourselves again, even after our value has been proven, albeit in an anecdotal sense.
The sine qua non of communication measurement is being able to show a provable and repeatable link between attainment of communication outcomes with business outcomes. This can be acheived either through a statistical analysis or through a qualitative assessment, but it must be measured.
I think that’s why, Mr. Wagner, the measurement mavens jumped on your case. As a profession, many of us come from backgrounds not exactly steeped in analytical behavior and deep quant theories. So our first step was to embrace measurement conceptually, and measure something — hence, ad equivs (which are still relevant in some cases), click-through rates, etc. (click-through rates compared to online sales? Ah!)
With apologies for the didactic polemic, relevant measurement is critical to our effectiveness — there surely are many ways to go about it — and we need to find out the best way to do it.
Sean Williams
Manager, Internal Communications
The Goodyear Tire & Rubber Company
Member, Institute for Public Relations, Commission on Measurement and Evaluation
InstituteforPR.org
March 5th, 2006 at 7:31 pm
Thank you all for an interesting dialogue. I, like Sean, am a member of the IPR’s Commission and I think he’s on to something: the first step in becoming truly professional is to become accountable. In my opinion, measurement doesn’t have to be “measurement guru” measurement (which is to say “expensive, sophisticated and, therefore, ‘not for us.”). I believe that it’s better to be partially right than totally in the dark, and while it’s desirable to link PR to meaningful business outcomes, sometimes it just isn’t possible, even with the introduction of the $300 dashboard. That being said, I feel that the best way to get started is to
* measure in small ways, whether you’re asked to or not
* reaffirm among everyone who is being measured that the measurement is about finding ways to improve rather than to dole out punishment. Every datapoint is a learning opportunity.
* set an example for others on the team by becoming your own “measurement guru” by taking advantage of free information from the IABC, the PRSA and the Institute for PR. Learn and apply
From surveys I’ve seen in the PR trades, a high percentage of PR people say they measure, but what passes for measurement by one may not meet the standards of measurement for another. The important point, I suppose, is that more and more PR people are becoming measurement-minded. As they begin to experiment and as they are exposed to a variety of research and evaluation approaches, they will naturally migrate towards more meaningful ways to clearly demonstrate and generate positive business outcomes.
March 5th, 2006 at 7:34 pm
Sean,
I think you make a good point: I’d rather be partially right than totally in the dark, and measurement in some form is better than no measurement at all. Industry surveys indicate that most PR people say that they measure, but what passes for “measurement” for one may not meet the standards for another. But even if people begin with simple inexpensive do-it-yourself tabulations of media coverage or web-based employee surveys, they will naturally migrate towards more meaningful measures linking PR and communication with business results.