How do I write this piece without making Peter Kim hate me? I guess I’m just going to have to give it a shot and hope for the best. It’s important to remember that this post isn’t about him. It’s about a piece of content.

None of this is personal. I even think like the guy. (We’ve never met in the real world, so I don’t know for sure.) I have a lot of respect for him and for what I think he does. (We’ve never worked together so I don’t know for sure either.) But I have to be honest, the 101 Examples of Social Media ROI list he published this week is crap seriously flawed. Here’s why: Most of these 101 “examples” don’t show ROI at all, “social” or otherwise. Either the title is wrong or the list is wrong for that title. One or the other.

Before I get into specific examples and illustrations of where I think the list fails, let me give you four basic problems I have with it as it stands today:

1. Many of the examples on it could potentially show positive ROI but – as presented – only reference selective gains from social activity and not actual, factual, empirical ROI. If that made no sense, that’s okay. Let me explain:

For something to be ROI, you need two ingredients: The cost of the activity and the gain from that activity. (That cost is the investment. The gain is either revenue or cost savings.) It’s math. Really really really simple math. ROI is an equation and it generally looks like this:

(\$ Gain – \$ Cost) ÷ \$ Cost = ROI

or

(\$ Revenue – \$ Investment) ÷ \$ Investment = ROI

(You can also multiply the result by 100 to get yourself out of the decimals, but that’s a personal choice. You can do that in your head.)

Anything that isn’t the result of the ROI equation is not ROI.

Note that a gain is just a gain,like cost is just a cost. Neither gain nor cost is ROI on its own. Ever. Not in any known universe.

Put another way, bread and ham may individually be part of the ham sandwich equation but ham alone is not a ham sandwich. Ham is just ham. The problem we face today: This list pretty much mistakes ham for a ham sandwich. Good thing it was free or we would all be asking for a refund (or a word with the chef).

Take this example:

61. Paramount Pictures. #Super8Secret Promoted Trend created a tremendous spike in conversations: Tweets of the hashtag reached nearly nine million impressions in less than 24 hours and mentions of the movie skyrocketed to more than 150 per minute. Receipts for the sneak preview exceeded \$1 million, and Paramount said weekend box office surpassed expectations by 52%. (Twitter, 2011)

Cool story, bro. What was the cost of the campaign?

Yes, this is an example of a successful use of social media (through a “promoted trend” media buy). Awesome. But where’s the bit that compares the \$gain and the \$cost? That would be an ROI example. This isn’t.

What’s sad is that there is probably an ROI piece hiding in the background but instead of focusing on that, the example dishes out a healthy helping of random gain data: Impressions. Mentions. Tweets. Retweets. Sales too, which is nice but no cost data… so thanks for playing but no. Without the cost piece, you don’t have an ROI example.

Your example needs to include this information or it doesn’t belong on that list:

(\$ Gain – \$ Cost) ÷ \$ Cost = ROI

Tip: If you can’t measure ROI or adequately prove it in this instance, that’s okay. Just don’t add it to a list of ROI examples.

(Speaking of proving cause & effect, let’s not forget that Super8 was a well anticipated \$50M summer fare from director J.J.Abrams and producer Steven Spielberg. Not exactly a grass-roots indie phenom that would have flopped without a promoted trend on Twitter. Let’s not go crazy over the role that social media really played in opening weekend ticket sales. A little perspective goes a long way.)

More examples of this disappointing absence of actual ROI metrics later. In fact… almost the entire list suffers from this single basic flaw. But hey, at least this type of example makes the effort of including at least a portion of the data that goes into an ROI discussion. Not all examples on the list do.

2. Many of the examples on the list don’t even reference financial gain at all, let alone ROI. I list more later in the post but these will get things started:

“68% of respondents said they were “much” or “somewhat” more likely to purchase post-project.” (Subaru. 80.)

“32,000 video views, 25% regular return visits to the site, and average of almost seven minutes spent on the site per visit.” (UPS. 96.)

Community drove a +20 NPS increase.” (Sage Software. 69.)

“58% higher engagement rate than people coming in from other channels.” (TurboTax. 91.)

These are very cool little successes, great things to celebrate and be happy about, but as valuable as they may be they are not ROI. Not one part of any of those numbers even fit in the ROI discussion. At least other examples on the list make an effort to list one element of ROI: Money saved or money earned. These don’t. Sorry but that’s a little perplexing.

Here’s an example of my own to illustrate how far these examples are from ROI: I love carrot cake and when people compliment me on my impeccable taste in carrot cake, that isn’t ROI either, no matter how much of those interactions happen online. I could call it ROI and score the number 102 spot on the list, thus:

102. Olivier Blanchard. Increased engagement with carrot cake enthusiast community by 37%. (The BrandBuilder Blog, 2012)

Except… no. It doesn’t work that way. Just because something is a success doesn’t mean it qualifies as ROI. Did my example mention that I even sold carrot cakes? Did I factor in the cost of making them or selling them online? Did I save money in any way by talking about carrot cakes with my twitter friends? Nope, I didn’t think so either.

Again, your example needs to include this information or it doesn’t belong on a list of ROI examples:

(\$ Gain – \$ Cost) ÷ \$ Cost = ROI

3. Some of the examples could have been bunched into one but legitimate examples were somehow omitted. Case in point: Cerner’s three examples (15, 16 and 17) are really one program / one example, but IKEA somehow didn’t make the list. (For more details on that particular program, click here.) Maybe scratching Giffgaff (32.) and replacing it with IKEA would have made sense? But okay, I’ll back off from this particular point. Lists tend to be incomplete. Someone always gets left out and sometimes you have to stretch yourself a little thin to get to the magic number. It’s no big deal.

4. Because of the source (Peter is well respected in this industry as far as I can tell), a lot of people will naturally accept this list as fact. It will become a template to be shared and passed around and referenced for the next couple of years. When marketing execs and digital agencies look for examples of ROI in social business, they will pull this thing from the Googlenets and use it as a resource for all sorts of things: Training of new social business recruits, client pitches, presentations at conferences, etc. They will do so without questioning the validity of the information they are not only ingesting but also sharing because they trust that Peter vetted the list before publishing it. That’s the unspoken contract of being a respected leader in the social business world.

Except… what if this one time, the information wasn’t properly vetted? What if much of it wasn’t even properly presented (using the right metrics, for instance)? Or what if the title is so wrong for the actual list that you end up confusing “value” with ROI for another 3 years as a result? Then what? No thanks. We can do better.

If you have 10 minutes to really get into it, read on. If not, you get the idea. (By the way, the list isn’t all bad.) Feel free to skip ahead to the end all the same.

* * *

Let’s look at a few of these examples a little more closely.

We’ll get to more obvious cases of “no, this isn’t ROI at all” a bit later. I want to start with some of the more subtle “maybe this could be ROI” examples first because a) they’re tricky and b) they illustrate pretty well some of the common traps people fall into when trying to establish ROI too quickly:

1. Aflac. Community drove online payments increase of 3% led to \$95,000 in savings. (Lithium Technologies, 2011)

Q: What’s the problem with this one, Olivier? It looks legit to me. What’s your deal?

A: Yes it does look legit. And it might be. But do we know anything about other activity from Aflac that might have contributed to that 3% increase in online payments?

Could a concurrent email or advertising campaign have triggered a significant portion of that shift? Could the addition of a flyer in the mail to existing customers prompting them to make online payments have been the real cause of the shift? We can’t attribute the success of “the community” until we have ruled those out. If we know for a fact that this was 100% the result of community engagement, great. Roll on. If not, we need to find out before we high-five the community management team.

Lesson #1: Assumptions are dangerous and attribution is tricky. If you are going to present an ROI example, make sure it is rock solid. Don’t assume that social business was the biggest (or sole) cause of your success.

A better way of presenting this one would have been to maybe connect the 3% lift in online payments to the \$95,000 in processing costs (context here would be nice so we know how the two might be connected). Tying these metrics to a specific campaign or activity on social channels wouldn’t be a bad thing too. Connect the dots a little bit: +3% in online payments isn’t ROI unless it results in \$x savings. None of it is an outcome of social business unless you also show how “the community” helped you get there.

Not saying this isn’t a potential ROI win, but as presented, we can’t know for sure. Not yet. We’ll give that a cautious MAYBE. Just watch those assumptions though.

* * *

2. Alberta Common Wealth Credit Union. Blog, YouTube, Facebook – 2 million impressions, 2,300 new accounts, and \$4 million Canadian in new deposits. (Forrester, 2008)

First, scratch the 2 million impressions bit. It’s a distracting metric and not super reliable (or even relevant to this discussion).

\$4 million in new deposits sounds like a great outcome for the program though. Here are the three problems with that:

Assumptions again: How do we know that these 2,300 new accounts and resulting \$4 million in new deposits were tied to the social media program (Blog, Youtube, Facebook) and not a combination of social and other factors (traditional marketing, advertising, PR, etc.). Can ACWCU realistically assign the 2,300 new accounts and \$4M in net new deposits to the social media program?

If the answer is yes, great. They’re on the right track. Time to back that up. Show me how that happened.

If the answer is no, then we have a problem right off the bat. Remember that thing about assumptions.

What about costs? What was the cost of the program? This example (and many others) don’t mention cost at all. They only mention gains. The ROI equation also factors in costs.

Here’s why this is kind of important in an “ROI examples” discussion: if the program or campaign cost \$4,000,001 and the net new deposits amounted to \$4,000,000, then your ROI was actually negative. Just sharing the gain from the campaign or program doesn’t give us any idea of what the ROI actually was.

Lesson #2: Don’t confuse ROI with gain. ROI is the ham sandwich, not just the ham. (Google the ROI equation, print it and tape it to your office wall. Before you tag something as ROI, make sure it fits the definition of ROI.)

No benchmarking: What the example doesn’t tell us is what the time period for this gain was, and how the credit union normally trends for similar time periods. What if ACWCU usually sees the same amount of new accounts and deposits for the same time period even without social media? Say that ACWCU saw 2,300 new accounts for the exact same period preceding the start of their social media program? Wouldn’t that mean that the social media program might have had no impact at all? You have to factor in time frames and set up benchmarks before you can weigh gains before and after the launch of a program.

Result: As presented, we have no way of knowing if the program perpetuated a trend or brought in new business above and beyond normal performance trending.

Lesson #3: Without adequate benchmarking, your ROI “reporting” is incomplete and doesn’t stand up to scrutiny.

File that one under MAYBE. (As presented: An incomplete report of gain but not an example of ROI.)

Way too many of this kind of anecdotal “example” on this list to make me comfortable with it. Sorry.

* * *

8. Blendtec. Viral videos increased company sales +700%. (Barnraisers, 2010)

That one actually does stand up to scrutiny. BlendTec’s hilarious videos (and live demos at trade shows) a) became such a hit and b) demonstrated the effectiveness of the blenders so well that orders for the blender increased almost overnight.

The reporting here is still pretty incomplete though: 700% over what time period? What else could have caused the increase? That’s a gain but not an ROI figure: What was the cost of the program vs. that 700% net gain in sales?

File that one under YES: ROI but with reservations. (As presented: another report of a successful gain but not an example of ROI.)

I really wish the legitimate ROI examples on this list actually focused on ROI instead of using disjointed metrics.

* * *

10. Bonobos. Exclusive sale on Twitter generated 1,200% ROI in 24 hours on promoted tweet. (Twitter, 2011)

First, proceed with caution if the list is about Social Business and you are just talking about a one-time media buy on a social channel. Social business is a little more elaborate than buying the odd promoted tweet for a one-day promotion.

Second, we have absolutely no idea how that 1,200% ROI figure comes from. What is it based on? Could the figure erroneously reference a 1,200% increase in sales rather than ROI? As presented, we don’t really know. Red flag.

Third (and perhaps most important) we have no idea what the cost of that promoted tweet was in relation to the gain in net sales.

Knowing nothing about this one, I want to give it the benefit of the doubt. Filing it under MAYBE. (I can’t believe I am being so nice. This would never pass muster during a legitimate business audit.)

* * *

13. Burger King. Subservient Chicken video increased chicken sandwich sales 9% per week a month after launch. (Adweek, 2005)

Again: At a cost of…?

If the 9% increase in chicken sandwich sales amounted to less revenue than the cost of the campaign or program, then the ROI was negative. This example (like most on the list) mentions gain without factoring in cost. This is the list’s biggest problem.

Footnote: Subservient chicken wasn’t just a social media campaign. Subservient chicken was an advertising campaign with interactive digital components. This is very different from a business like Best Buy or Ford engaging with people via social channels to grow mindshare, improve the brand’s image and ultimately increase preference in the minds of x% of car buyers. When looking at this type of hybrid model of social and traditional media, you cannot legitimately talk about the ROI of “social”.

Lesson #4: When a campaign (note my choice of the term campaign and not program) is as much social marketing as it is traditional marketing, you cannot attribute its success to “social media” or even “social business.” An advertising campaign, even with social channel components is still an advertising campaign.

Effective, sure, but still just advertising.

File that one under a cautious and suspicious MAYBE. (As presented though: No ROI was actually demonstrated here. Value: yes. ROI: nope. Again.)

Let’s move further down the list.

* * *

Let’s leave the gray area of “maybe” for a minute and look at a few examples that don’t fall anywhere near ROI (as presented):

15. Cerner. Community resulted in 13% fewer customer support issues logged. (Jive Software, 2011)

16. Cerner. Community resulted in 70% decrease in internal HR issues logged. (Jive Software, 2011)

17. Cerner. Community resulted in shorter approval cycles for writing technical documentation, from 2-6 weeks to hours or days. (Jive Software, 2011)

19. Charles Schwab. Online community drives 56% increase in Gen X customer base versus year ago. (Communispace, 2007)

20. Cisco. Community deflects 120,000 support cases each month. (Lithium Technologies, 2011)

24. Electronic Arts. EA was 2nd UK brand to use promoted tweets and trends to promote FIFA 12 video game. Trend engagement level was 11%, well above Twitter’s average ‘benchmark’ for trends, of 3% to 6%. Promoted tweet engagement averaged 8.3% over two-week campaign vs. Twitter benchmark of 1.5%. (Marketing Magazine, 2011)

25. Elsevier. Wiki drives 80% reduction in interdepartmental e-mail volume. (Socialtext, unkn)

28. FICO. Community: 850k customers served, resulting in 10% improvement in call deflections annually. (Lithium Technologies, 2011)

30. FONA International. Wiki eliminated almost 50,000 e-mails a year from one specific process. (Socialtext, unkn)

32. giffgaff. 100% of questions answered by community members in average time of 93 seconds. (Lithium Technologies, 2011)

34. Hershey’s. House party: 10,000 parties, reached 129,000 people, and say their campaign was seen by 7 million people. (Forrester, 2008)

35. Honda. Friending Honda campaign increased Facebook fans from 15k to 422k, generated over 3,500 dealer quote requests. (RPA, 2010)

36. HP. More than 4.6 people have told HP that the forum solved their support issues which HP says makes customer happier and saves the company millions in support costs. (Forrester, 2010)

42. Intuit Quickbooks. Business owners engaged with rated ProAdvisors 555% more often than unrated counterparts. (ratings and reviews). (Bazaarvoice, 2011)

No ROI in any of those examples whatsoever. There are more but I will let you find them all on your own.

Lesson #5: If it isn’t a \$cost vs. \$gain equation (or whatever currency you need it to be), it isn’t ROI. Customer base, leads, referrals, links, clicks, retweets, HR issues logged, email volume, estimates of future sales, deltas in NPS, quote requests, parties reached, impressions, engagement, etc. = not ROI.

Note: Too bad HP (36.) didn’t lead with the “saves company millions in support costs.” That looked like a legitimate ROI example. (Right company, wrong metrics to illustrate the ROI piece.) It matters that 4.6… wait. 4.6 people?

Maybe it was 4.6 million? Or 4 out of 6?

Anyway, whatever the number is, it matters but it is irrelevant to the ROI discussion. What would have been relevant would be how many millions in savings HP enjoyed as a result: The cost of implementing and managing the program vs. the \$x million savings would have been a perfect way to illustrate ROI here. Missed opportunity #36 on the list so far.

Speaking of how to properly present ROI “examples,” here are a few quick tips on how to turn these examples into legitimate ROI stories:

It would have been great for the three Cerner examples to talk about actual cost reductions from the drop in customer service and HR issues, for instance, but they didn’t The metrics used had nothing to do with ROI. Financial gains (either via revenue or cost savings) were never mentioned. The cost of implementing and managing the program(s) was also never mentioned. Why? Those are far more relevant metrics than the ones presented.

Same with Elsevier: An 80% reduction in email is great but what is the impact on operational costs? That would be a potential ROI story.

Honda (35.) would have a great ROI story to tell if it could show the net number of sales from those 3,500 dealer quote requests and then scrubbed from that list every buyer who was going to buy a Honda anyway, regardless of the company’s social media activity. Presenting the example with “likes” and “dealer quotes” as the two principal KPIs (key performance indicators) instead of net sales (for example) puts the example squarely outside of a legitimate ROI discussion.

Intuit is another example of a company listed here with a legitimate ROI story to tell, but the description references a KPI that has nothing to do with ROI whatsoever. “555% more engagement resulting in net new \$… versus a cost of \$…” would have scored a bullseye. “555% more engagement” alone doesn’t.

Is it too much to ask for a list of ROI examples to actually use cost vs. gain numbers? As in… the actual ROI equation? Because that would be simple, clear and nice… and relevant. Instead of…

19. Charles Schwab. Online community drives 56% increase in Gen X customer base versus year ago.

… try this:

19. Charles Schwab. Online community cost: \$X. 56% YoY increase in Gen X customer base attributable to online community resulted in net new revenue of \$Y FY2011. ROI: \$Z.

Simple. That’s how it’s done.

Perhaps there is an ROI story hiding somewhere in the background of every single example here. In fact, there probably is. But these examples, as presented, don’t talk about ROI at all. They reference non-financial gains without establishing any link whatsoever to ROI. So… Sorry, that’s a big zero on all of those.

Filing these under: NO ROI ANYWHERE (except for the vague afterthought in number 36).

My thinking: Far too many of these on this list as well.

* * *

22. Dell. @DellOutlet on Twitter generated \$2 million direct sales, influenced \$1 million addt’l (2007 – 2009). (Direct2Dell Blog, 2009)

Yes. Tweets linked to offers were tracked and a direct path of tweet-to-purchase was clearly established. Empirically.

File that one under YES: ROI. (But it would have been nice to see it as an ROI example and not as just another example of gain.)

Cost of program vs. \$ in net sales from the program. Simple. Another missed opportunity to demonstrate ROI properly.

Moving on…

* * *

27. Epson. Reviews drove 98% higher revenue per visitor for Epson. (Bazaarvoice, 2011)

First, I have absolutely no idea how one leads to the other. How do we know that “reviews” drove higher revenue per visitor? Show me how you came up with that figure.

Second, what does that have to do with ROI? (Gain from reviews – Cost of reviews) ÷ Gain from reviews = … oh wait. What was the cost of those reviews again? #Fail. Value: Yes. Correlation between A and B: Maybe. ROI: Nope.

Sorry but I have to file this one under NO. Interesting but not ROI.

* * *

Dancing back into ROI territory now. (I still feel guilty about pointing out the problems with this list.)

37. IBM. developerWorks community saves \$100 million annually from people who use this resource instead of contacting IBM support. (Forrester, 2010)

38. IBM. Crowd-sourcing identified 10 best incubator businesses, funded for \$100 million, generatiung \$100 billion in total revenue for a 10-to-1 ROI with a 44.1% gross profit margin. (Barnraisers, 2010)

Now we’re talking. ROI can come from cost savings, not just net new revenue. Well done, IBM.

Filed under YES: ROI.

* * *

45. Jewelry TV. Customer reviews boost mobile sales by 30% (ratings and reviews). (Bazaarvoice, 2011)

Aside from the obvious problems already encountered with previous examples, this one introduces us to a new one: The 30% boost in mobile sales. Is this 30% net new sales or simply a shift from non-mobile sales to mobile sales? Whether someone buys from a mobile device or their land line, is there really a difference? Does it have anything to do with ROI?

53. Mattel. Despite product recalls, online community helped support Q4 2007 sales increase of 6%. (Forrester, 2008)

How do we know that the online community helped support a Q4 2007 sales increase of 6%? isn’t it more likely that back in 2007, advertising, product placement and good PR might have been more responsible for that 6% increase than an online community?

Also, 6% versus what? Is this YoY or QoQ? Was it normal for Mattel to expect 6% growth in Q4 of 2007 with or without an online community?

Too many unanswered questions = too many assumptions.

Filing these and others like them under NOT SURE WHAT THAT WAS. MAYBE.

Another reason why benchmarking matters. Just throwing numbers around without establishing a context for them doesn’t really tell you anything. Data can be manipulated to tell wonderful stories when no one is there to ask hard questions like “prove it.”

* * *

I want to end on a positive note, so here are several examples that either have potential or are clear examples of ROI (in no particular order):

11. Bupa. Community drove £190,000 savings through collaboration, online events. (Jive Software, 2011)

100. Vistaprint. Community tracked \$30,000 in social revenue in 2009. (Lithium Technologies, 2011)

23. Domino’s Pizza. Foursquare drove 29% pre-tax profit through promotions. (Barnraisers, 2010)

71. SAP. Community drive 5% increase in partner sales. (Jive Software, 2011)

57. National Instruments. Community resulted in 46% of all support questions answered by peers instead of support. (Jive Software, 2011)

84. TomTom. In one month, community handled 20,000 cases resulting in \$150k of savings. (Lithium Technologies, 2011)

65. Precyse Technologies. \$250,000 savings in crowdsourcing new product design. (InnoCentive, 2010)

92. TVG. Community members spend 36% more than average. (Lithium Technologies, 2011)

67. Rhapsody. 50% decrease in support costs and 53% decrease in weekly support contacts via sCRM solution. (GetSatisfaction, unkn)

60. Orange. Listening: saved a few million euros in support costs and helped avoid several potential PR problems. (Forrester, 2010)

75. Secret. Among women viewing the video, 57% said their impression of the Secret brand improved and purchase intent among women who participated on Facebook went up by 11% (33% for teens). Clinical sales increased 8% despite a 70% decrease in TV support. (Forrester, 2010)

85. Toshiba. Saved \$213,000 by adding online component to 5 events, doubling attendance. (Jive Software, 2011)

95. University of London. Internal social network allows students to collaborate remotely, expected to deliver future savings in the region of £300,000 per year in print, courier and administration costs alone. (IBM, 2008)

While examples like Secret (75.) and SAS (71.) require you to make leaps of faith (as presented) and don’t actually give us ROI data (not just gain but relative cost of the new activity vs. traditional spend), you can see an ROI story forming in the background. It’s still vague but you can tell it’s there.

Let’s file those in the “PROBABLY ROI (if we dig a little more)” folder.

Examples like Orange (60.), Precyse Technologies (65.) and TomTom (84.), on the other hand, are cut and dry: The cost savings are empirical. You can tie the cost of the activity to the financial gain to the company.

We’ll file those in the “YES: ROI” folder.

Special mention for actually listing both gain and cost:

88. TransUnion. Estimated \$2.5 million in savings in less than five months while spending about \$50,000 on a social networking platform. (Socialtext, 2009)

If only all 101 had done that.

* * *

(If you skipped ahead, pick up the post here. You’re almost done.)

Conclusion:

If you look at the list from the perspective of “these are 101 examples of where social business has benefited or added value to a company” then it is solid. Kudos to Peter and his team. Great title, lots of value there, please share with the world. Just make sure you scratch out the title or petition Peter to change it.

If you look at the list as a collection of “101 examples of social business ROI,” then the list is almost entirely wrong. Back to the drawing board. Sorry. It doesn’t work.

I don’t want to just point out the flaws without offering Peter a way to fix it, so here are the only two options:

1. Change the title to something along the lines of “101 examples of successful Social Business campaigns”. (Remove the ROI bit from it if you aren’t actually going to focus on ROI.)

2. Include actual ROI numbers for each of the 101 examples. (Those can just be the cost and the gain from activity figures. Real simple.) Even if some of those ROI numbers turn out to be less than impressive, the list will still be factual and valuable.

Oh, and 3. Include IKEA. It deserves a nod.

* * *

I almost forgot…

Lesson #6: Ask the hard questions. Don’t assume that information (or insight) from anyone in any industry that touches marketing in any way is accurate. Not even mine. Put everything through your own stink test. Use your noggin’. Challenge everything that raises a red flag. Learn the definition of business terms too. They matter. Worth keeping in mind next time a list like this pops up (and there will be more like it).

Or your could just Google “R.O.I. calculation” for crying outloud. Every kid with a lemonade stand grasps that math. Why can’t social media gurus? It boggles the mind.

Cheers,

Olivier

PS: Everything in this book could also be dead wrong. It could all be pure BS. Scrutinize it as well. I’m not immune to the occasional wrong conclusion either. You never know.