My latest measurement is “slices.”

We did a TV show about bagels - Since some of the people in this thread are Canadians, Breakfast Television, CITY TV, a remote with Jennifer Valentyne — and the next day learned that the shop owner ran out of Montreal smoked meat that afternoon, after Jennifer ate a smoked meat sandwich on camera around 8:15 in the morning.

Or cream cheese. That almost disappeared, too.

Plus, the client gave me a blogo’d cap, and over the next few days several stangers told me they’d seen the company on television, and then gone to buy bagels.

Or dog treats — we got talking to a store owner (sells pet stuff and we were in to buy dog treats) about publicity and she mentioned she’d seen a TV show that was so compelling her husband drove 20 miles to buy stuff to take to his family. “Our” bagels.

Or, as measurement, reduced leftovers. The client takes the leftover bagels to several organizations that feed less fortunate folks, and the numbers of free bagels dropped so much that he started baking just for them.

Or, as measurement, the cash register.

The good part about some publicity is that you can isolate the effect, as per a retailer on television. Harder when there’s a coinciding advertising program, a big public exhibition, free tastings at Main Street and First Avenue, and 20 percent off coupons in the stores.

BAK

Christine

Christine Nyirjesy Bragale, ABC

P.S. to Wilma: we have a valuation guide at http://www.goodwillpromo.org that might help!

Congrats on wanting to make your measurement program a solid one.

I think we often confuse various measurements when it comes to media.

First, we can measure or evaluate the process of our media relations efforts: are we operating in the most efficient way possible; do we have the tools/software we need to run an effective media relations shop; is our approval process getting in the way of our media plan…those are a few process measures.

We can also measure/evaluate output: are we sending out the right information to the right media in the right way at the right time for the right reason(s). Too often, as you know, there are requests (demands?) to “get some media coverage” on something that just isn’t suited for media. Surveying editors (as part of an omnibus research effort) will help answer questions about both your output and your relationship with media representatives.

We can finely dice measuring output as it relates to outcome: did those news releases and pitches result in media coverage — where, when, how, above the fold, etc.

And then we can measure bottom line outcome: did people write their congressional representative? Did more donations come in to Goodwill? Did more people bring items to Goodwill because they know how the money is used? Did the toll-free telephone number light up after a certain media push?

I understand the pressure(s) that media relations professionals have (or believe they have) and I believe that we rely too much on formulas and quantity to help alleviate that pressure.

You’re already pushing the needle towards awareness and will need good polling before and after a media campaign to know if you’ve increased the awareness on your topic. Until then, keep learning and asking…this is a community of individuals who understand where you are and want to help.

Best wishes,

Wilma Mathews

P.S. Thanks for your support on the tax issues related to non-cash donations. Some minor restrictions have been put in place, as I noted when working on my material yesterday to send to my accountant. As usual, the new language is just ambiguous enough to cause problems.

I think the short answer is that you’re off to a wonderful start, so congratulations. What I find really encouraging is that you’re clearly looking on communications research not strictly as post-campaign evaluative, but also pre-strategy formative. Indeed PR can and should be a management science, not just a dark art tactical afterthought.

The indicators that you’ve listed could be tracked as part of an on-going media content analysis project. You might consider reaching out to a vendor in that business. (In the interest of total transparency, we are in that business, largely in Canada, but there are a number of other excellent providers in the U.S.)
Content analysis suppliers can track literally hundreds of variables for you and they will work with you to find those most meaningful for your organization. You might also want to look at variables such as hot button issues, executive profile, competative/peer organizations, consumer mention/sentiment and a sort of framing variable that looks to categorize how the story mentioning your organization is framed/presented.

You hit the nail on the head with reference to ‘moving the needle’ and ‘awareness.’ You noted that you;d undertaken some awareness polling in 2004. Communications and marketing people in your organization may want to consider updating that research to see what progress you’ve made. Costs could be shared among all departments that would find such data useful. You might want to consider adding some basic media consumption questions to the survey such that, come results time, media content analysis results could be correlated with awareness results. To do that, it’s best to have a media content analysis supplier play some consultative role with you and your polling provider at the design stage.

Of course, all this begs the question of direct proof and causality and things like accounting for PR’s influence versus marketing, among others. We know that PR does not (nor should it) occur in a vacuum. And decades of academic research on the topic tells us that people just aren’t as easily and diretly influenced by communications efforts exclusively. That said, there is progress being made on statistical modelling employed and, at worst (or best depending on how you look at it), you may find it another useful analytical tool to add to your list of indicators. Not the end all be all, but another tool among many.

This would a be a bit trickier, but you might also consider looking at correlating donation activity during peak campaign times to the media content analysis. But, again the same uncertainty and debate as it relates to causality remains.

Food for thought.

Cheers
Alan Chumley
VP, Cormex Research
http://www.cormex.com

My measurement “program” (underline the quotation marks) is still an infant – but I’d like to know if you think my approach is off the mark or a decent start.

Short background: I’m at Goodwill Industries International, a well-known nonprofit organization. Bet you’re thinking it’s time to clean out your closet (please do!). My problem is that, according to a 2004 natinal (U.S.) public opinion survey, 88 percent of you don’t know that the revenue we earn from selling your donations in Goodwill stores funds job training programs that benefit hundreds of thousands of people each year. In other words, when you hear Goodwill, you know donations and thrift stores. But we want you to know that donations and Goodwill stores get turned into Goodwill programs that help the most vulnerable people in your community find jobs and earn a paycheck so they can support themselves and their families.

So, back to what I measure – As a media relations professional, my job is to get stories in the press (and, as a veteran journalist, to the person who said “above the fold is the journalist’s success,” I would say it’s the success of the communicator as well, because it’s got to be a solid, well-framed story to capture the journalist and the editor’s attention, and a lot of cleverness, patience, persistence and tenacity on the part of the communicator to shepherd the process). I do need to measure the things that show I’m getting stories in the paper. But I look at the totality of these things as indicators to see if I’m making baby steps in the right direction – that is, at some point will I be pushing the needle towards better awareness?
• Number of clips
• Tone
• Message inclusion
• “Mission” mention – i.e. articles about retail/donations that mention what they’re used for. Believe it or not, even reporters don’t care what we do with the stuff you give us! Or don’t think it’s important to share that information with you.
• Media markets (relevant to a specific outreach effort, for example, on a public policy issue, are we hitting markets of the legislators we want to reach?)
• Placement source – i.e. unsolicited reporter call, pitch from my office, etc.

There are also specific efforts where I can see audience reaction – for example, two years ago we fought against a proposal to limit your tax deduction for clothing donations. Aside from the positive legislative outcome, we also saw hundreds of random people log onto our web site and use a web form to send letters of support to their congressional representatives. I’d love to have the resources for a PR campaign which would involve something like a special 800-number so I would know everyone calling is responding to my campaign.

Other efforts – cause-marketing partnerships, advertisting, etc – will help impact mission awareness as well, but I see the above as a start in measuring the earned media piece of the pie.

What do you think, off the mark or a decent start? I welcome your comments!

Christine Nyirjesy Bragale, ABC

Thank you for your welcome.

Your last point about raising the bar is exactly on the money…and as professional communicators all of us need to develop still further the ‘political’ will to undertake good research and evaluation of the kind that enables us to understand genuinely just how successful (or otherwise), in exact terms, our campaigns and activities are.

To be blunt, there are still too many in the industry who simply don’t want to know… because properly audited results will not be flattering. If such people can persuade their client or manager to accept the proposition ‘if the campaign looks good, then it must be good’, then they will simply reply “why bother?” when the notion of proper research and evaluation is raised. In agency terms, the planners are at least as important as the creatives, if not more so. If a brilliant campaign fails to reach the required audience, all has been in vain.

If the industry truly welcomes the idea that payment should be by results achieved and not effort expended, then the quality and accuracy of work done can only get better and better. Clients and Executive Boards carry a lot of responsibility - and in their own interest too - to ensure (and provide budget for) robust evaluation solutions.

In closing, I will differ with you just slightly if I may on your point about generating ink and electrons! It seems to me there’s just the merest hint of a little complacency here… that any coverage is good coverage, and that anyone can do it!

Achieving exactly the right tone and content of coverage, in front of exactly the right audience(s) and at exactly the right time is far from easy and yet doing so is critical to the further steps of changing behaviour…and that’s exactly why detailed and accurate media analysis is indeed so important - as diagnostic, for quality assurance and in helping achieve the ultimate business result.

Your series looks to be admirable and I hope many will participate. Thank you for your excellent initiative!

Nicholas Grant
Chairman of the AMEC Board (www.amecorg.com)
CEO Mediatrack Research (www.mediatrack.com)

Thank you for your input and bringing AMEC to our attention.

Like you, I believe any measurement is a start. But I think too often we get a bit complacent and don’t focus on what sets us apart.

Ultimately, anyone with a reasonably interesting story can “generate ink” (although I guess we should now call it “generating electrons”).

I was motivated to do this series because I judged a number of media relations awards programs and was somewhat disheartened by the number of our peers who simply aren’t getting it.

Hopefully this series, with input from professionals like you and others, will help us all raise the bar.

As an IABC member for some years and as Chairman of the Board for the international Association for the Measurement and Evaluation of Communication (www.amecorg.com) I would commend all those with an interest to benchmark the AMEC URL, look at the AMEC College and join us in our activities. AMEC comprises more than 23 research companies from Europe, Australia and the Americas with a burgeoning individual membership too and runs training courses and a Quality Assurance standard to help promote best practice.

In terms of the discussion so far, I’d like to simply underscore a few basic principles.

Of course, behaviours and outcomes are ultimately the most important goal, but that doesn’t mean that interim processes are without value, far from it. The purpose of media content analysis is rather less to do with computing ‘an evaluation’, and rather more to do with providing a robust data platform which can then be used for many different purposes including performance management, audit, resource planning and testing hypotheses as to cause and effect.

AVE, crude multipliers, unreviewed interpretations of favourability and measuring column cms are all of very limited value. But each project should be set up to use the tools that are most appropriate. There is no silver bullet, and those that lust after one are missing the point. Successful communication of ideas and even product endorsments is too complex to be susceptible to a single algorithm.

The point about whether a story is above or below the fold is not about the effort and skill of the communicator (or the journalist), it’s about the likeley outtake by the audience(s) for that piece…for that’s the raw material the communicator then has to shape further as the issue progresses.

Eric, you asked about message inclusion. Some call it pull through or resonance. Often we recommend to clients that 3-5 of their overarching core reputational messages be tracked in their coverage. We look for those messages (or the spirit of them) and we look at whether they are present, present supported, present but countered (and by whom in each case). Fairly common among the other content analysis shops out there.

One point on moving toward outcomes (beyond the clump ‘o clips and the sales correlations). And agreed, PR is much more than about either. Beyond Katie’s assessment which is spot on, PR is about corporate reputation, trust, sponsorships and relationships–all of which can be measured.

While it’s not behaviour, it’s the middle ground of opinion. We’ve done some work correlating trends in media coverage and profile with public opinion polling data. Two challenges here. First, while the data often already exists, it resides with the clients and they can be reluctant to turn it over. Second, the media content analysis shops aren’t always brought in before the public opinion polling process begins. Ideally, the media content analysis shop would be at that consultative table so that, at the very least, certain medai consumption questions could be added to make the correlation work down the road easier and more relevant.

As many in the measurement business will say, and I recall hearing Katie mentioning this, or something akin, at the measurement summit, frequently there are all sorts of data that resides with the client that could be pulled together if loked at strategically and holistically. The challenge is which sources, getting access to them and what to do with it once you have it.

I agree with you and I believe you’ve hit on the source of the discomfort I’ve felt for some time with any form of line counting of any kind.

It really doesn’t take into consideration those other elements — conversations, networking and discussions — which ultimately lead to people developing attitudes and opinions (and perhaps expresses itself as behavior out the other end).

So how do we get to the point of examining outcomes, rather than counting inputs?

//eric