Yesterday I received an email from a major marketing firm offering a download of statistics to use in marketing and sales presentations. In the body of the email, the catch phrase that they used to get me to download their resource was:
Numbers don’t lie
The problem is, numbers can lie. And then when you go to interpret them, they can lie even more. You can make numbers pretty much say anything you want them to.
As I looked at the numbers in the presentation, I compared them to numbers from other studies that showed the exact opposite.Some of that contrarian research might have been from this same company, in fact. How does that happen?
Related Resources from B2C
» Free Webcast: Build Better Products by Identifying and Validating Your Riskiest Assumptions
When someone offers a study that tells you that
X% of social platform Y’s users click Z% of the time
you need to look more closely at the numbers. Ask yourself some questions:
- What were they really measuring?
- How were they measuring it?
- What was their sample/universe?
- What was the time frame of the study?
- Are their interpretations correct?
Remember, when studying behavior online, it isn’t like chemistry or physics. In the physical sciences we have things called “laws.” These exist because of testing that has been done, and the result is always the same under repeated trials, under the same conditions.
In the online realm, what we find aren’t laws. The conditions are never the same. Everything changes from moment to moment. This is why it’s so difficult to measure influence. It isn’t static. In chemistry when you combine two hydrogen atoms with one oxygen atom you get water. Every time. But your experience with posting things on Facebook, or writing a blog post, or tweeting, will be different. Every time.
When I look at data like this with my clients, I have to remind them that what they are seeing is big picture data from a specific period in time. It may or may not apply to their situation. In his recent post on Convince and Convert, Emeric Ernoult discusses this in terms of the discussion around the numbers relating to engagement on Facebook, particularly the idea that Facebook business page posts reach an average of about 16% of fans:
This famous figure of 16% has been repeated on all social media blogs so that it is now perceived as an uncontested truth by virtually all Facebook page administrators.
Of course, the reality is very different: Your posts don’t reach exactly 16% of your fans, but most likely a little more or a little less — sometimes a lot more or a lot less.
This 16% figure is the average of data from millions and millions of pages of very different types, it is a myth.
That’s the thing about data: it often represents an aggregate of information that may not be the same as my experience. If the data mentioned above includes data from the Walmart, Oreo, and Justin Bieber Facebook pages, well, that will color it’s relevance to my client who owns a small local business with only five employees.
In fact, even if you take data from your own experience over the past six-months, you can only use that as a guide for how you should move forward. Yes, it is hard data that is a snapshot in time of your performance, but there are no guarantees that things won’t change for the better or the worse. That’s why when you hear those commercials for financial products, they always add a disclaimer along the lines of:
Past performance does not guarantee future results.
This is not to minimize the importance of data. Just be careful how you throw those numbers around. For every study that says Facebook is great for engagement, leads, or whatever, there is a similar study that tells you the opposite. You need to read the numbers, understand where they came from, and interpret them for yourself. Only then can you decide whether they are something that applies to your very specific situation.
Take numbers at their face value, and then learn how to interpret them and determine how useful they are to you. Just because someone quotes a study, doesn’t mean it’s true. Or at least not true for you.