Measurement is such a tricky thing. Whether you’re trying to brew your morning coffee or analyze a full-scale marketing program, there are more ways to do it wrong than to do it right. But don’t let that scare you away from measurement. No matter how many terrible cups of coffee I make, I’m diligent. Mostly because I need the shot of caffeine and somewhat because every once in a while, I actually get it right.
Now, before I go any further let me point out that no one is ever going to mistake me for a measurement expert. I can’t geek out on statistical anomalies, p-values or standard deviations any more than I could lace up a green apron at Starbucks and make you a tasty Grande, Non-Fat, No-Whip Cinnamon Dolche Latte (My wife’s usual).
Which is fine because we have baristas and statisticians for complicated matters like that. My advice is more geared toward that solid cup of morning joe or those every day decisions we make based on “the data” that we so frequently torture until it tells us what we want to hear.
With that disclaimer out of the way, here are my favorite measurement flubs:
The N of 1 - ”We tried that, and it didn’t work.” Whenever someone says that to you, the first thing you should ask them is this: How many times did you try it? In most cases, that someone will respond with “once.” That’s a bad answer, by the way. This is a dangerous and unscientific way to make decisions.
The Pile On - This happens when you introduce too many variables into your experiment, and you can’t clearly evaluate what’s working and what’s not. Throwing one more log on the fire can quickly burn down the forest of insight you’re seeking. On a related point, just because you can measure something doesn’t mean you should.
The ROI Trap - Sometimes tactics can be wildly successful in “influencing” specific actions and buying decisions without getting credit for “closing the sale.” If you focus myopically on ROI, you may conclude a tactic is not worthwhile, despite it being mission critical to your success. More often than not, it is the combination of marketing techniques that do the trick.
The Blink - Just because it works for Malcolm Gladwell doesn’t mean you should apply it to measurement. Sometimes people approach measurement like this: They plant vegetables in the garden, and then dig them up the next day to see if they are growing. In our instant gratification society, we are used to having our questions answered immediately. Don’t pull the plug at the first sign of trouble. Instead, allow time to capture enough data to make informed decisions.
The Gut Check - This is when someone blatantly disregards what the data shows, because it doesn’t validate his or her gut. Our field is both art and science. But sometimes we default to our gut, even when there’s ample proof that it’s wrong. We slice, dice, twist and turn until we get the numbers to tell the story we want.
The Missing Link - Sometimes we fail to identify how we’re going to measure success in the first place. And then halfway through, we realize we don’t have all the data we need to truly evaluate the campaign. It’s obviously important to spend time up front considering how you’re going to measure. But campaigns have launched without this forethought.
In summary, to effectively measure:
- Always plan ahead.
- Avoid jumping to conclusions.
- Don’t measure impact too narrowly.
- Don’t try to measure EVERYTHING.
- Definitely don’t try to measure everything at once.
- And finally, it’s one heaping, but rounded scoop for every two cups of water. Turned to the strongest setting. Pour, drink and enjoy!
I would love to hear about other common missteps to measurement. Mostly so I don’t have to make them myself. Any tips on coffee brewing are welcome as well!
Great article Heath. I just want to expand on your third bullet about the ROI trap. ROI is about generating more profit back than you spend so that goal needs to hold for all of our marketing initiatives.
The “trap” is more about measuring sales outcomes vs. non-sales outcomes, which requires a balancing act. I would split this into two parts:
1. Your marketing should have some incremental contribution to sales even if it is not easily tracked. Either set up measurements to capture incremental sales or have a very good understanding of how other engagement outcomes relate to sales.
2. The flip side is don’t depend on “feel-good” measures that are easily tracked and show marketing impact, but have little or no influence on sales.
Getting a reasonable assessment of your marketing contribution at least gives you insight to compare tactics and improve effectiveness.
And I wasn’t allowed to make the coffee until the Keurig machine showed up. When are we going to have marketing measurements be that easy?
Jim
Jim, thanks for the feedback and the additional points! I agree that it’s a balancing act. It’s also easy to roll the other direction and not give campaigns enough credit for enhancing brand equity and strengthening positive perceptions in the marketplace, etc. Really tough to measure but important. If they ever create a Keurig machine for marketing measurement, I’m all in! Which begs the question of why there’s not one in my kitchen for my morning caffeine already!