This Is a Test…

In Sep­tem­ber, I out­lined the five strate­gic con­sid­er­a­tions for suc­cess­ful opti­miza­tion cam­paigns. One of these steps is crit­i­cal yet over­looked by count­less mar­keters: test­ing. I gave the exam­ple of a gym membership—if you don’t lever­age the gym (or your test­ing capa­bil­i­ties) in a con­sis­tent and dis­ci­plined fash­ion, it really doesn’t deliver any sort of value.

The most suc­cess­ful orga­ni­za­tions are div­ing into sys­tem­atic test­ing built on a well-developed strate­gic plan, crys­tal clear method­ol­ogy, and a uni­ver­sal goal align­ment across stake­hold­ers. This starts with a rock solid method­ol­ogy devel­oped and agreed upon by all. What’s involved? For starters, get­ting those key play­ers on board and solic­it­ing feed­back from all sides to ensure a deeply col­lab­o­ra­tive begin­ning. From there, it’s essen­tial to fully invest in and carry out all core steps of the process, then “wash, rinse, repeat, and improve” the next time around.

Hypoth­e­sis Development

It’s just like the sixth grade sci­ence fair—you need a hypoth­e­sis. What do you think will hap­pen when you test this mes­sag­ing or that pro­mo­tion or the inte­gra­tion of a new pur­chase path? Although the sci­en­tific method for devel­op­ing a con­ver­sion opti­miza­tion hypoth­e­sis is all well and good, it’s also essen­tial to keep an eye on inputs—think qual­i­ta­tive and quan­ti­ta­tive data, tacit knowl­edge, and of course, your gut as a marketer—to put forth an edu­cated guess as to what’s next. Then prove it. Or dis­prove it.

Test Design

Recommended for YouWebcast: Why, What, and How to Do Social Selling

When you’re A/B testing, the approach and design can often seem like no-brainers—most likely, you’re test­ing your cur­rent site, or some extended ele­ment of it, against a new visual, mes­sage, pro­mo­tion, or con­tent piece. A/B test­ing, in gen­eral, is most effec­tive when you’re look­ing at two ver­sions of one variable—color, mar­quee image, pro­mo­tional mes­sag­ing, and so on. Keep an eye on sam­ple sizes to ensure the num­bers com­ing back are both solid and sta­tis­ti­cally sig­nif­i­cant, and be sure you’re col­lect­ing and assessing.

For mul­ti­vari­ate test­ing, site traf­fic should be evenly dis­trib­uted across all ver­sions being tested, so make sure you’ve got a good amount of traf­fic. Oth­er­wise, A/B test­ing is likely the bet­ter choice to ensure con­crete results in a rea­son­able amount of time.

Defin­ing Success

Set­ting suc­cess met­rics is not only an essen­tial part of the test design process, but a crit­i­cal step on its own. Once a hypoth­e­sis has been defined, you need to define suc­cess bench­marks for your test­ing. What will make A more suc­cess­ful than B? Is it sim­ply a lift in sales? Higher con­ver­sion met­rics? Engage­ment? Larger orders? Or some­thing entirely dif­fer­ent? Your test­ing tool will record vari­a­tions shown to the site vis­i­tor and help you get the com­plete pic­ture and deter­mine what worked and what didn’t.

As with every­thing else, the stake­hold­ers must agree from the get-go on what “suc­cess” means through­out the test­ing process and beyond. If you don’t under­stand how your orga­ni­za­tion defines a suc­cess­ful ven­ture, you won’t know if you achieved any objec­tive and, of course, get­ting the resources allo­cated for future test­ing will be challenging.

Exe­cut­ing a Test

Now do it! Exe­cute the test and watch your met­rics, but be care­ful not to con­clude any­thing too early. Sta­tis­ti­cal sig­nif­i­cance and con­fi­dence are essen­tial to the test­ing process, and too few engage­ments and con­ver­sions sim­ply don’t tell the full story. Like­wise, be sure you’re keep­ing an eye on offer per­for­mance by indi­vid­ual seg­ment. Look­ing at all the traf­fic at once can lead be mis­lead­ing. A pro­mo­tional or spe­cific out­reach may have fallen short in one crit­i­cal mar­ket, didn’t con­vert on a cer­tain device or browser, or sim­ply failed to engage female shop­pers. It’s not enough to give a thumbs up or thumbs down—dive into the seg­ments and their responses and see how you can improve in the next go-around.

Beyond seg­ment­ing and scale, it’s also crit­i­cal to care­fully out­line the dura­tion of your test in the ini­tial plan­ning stages. This should be the first of many tests fol­low­ing the wash, rinse, repeat, and improve for­mat, so keep a close eye on it through all stages of the roll­out so you can con­tinue to enhance down the road.

And remem­ber, although your gut is essen­tial to the hypoth­e­sis stage, it shouldn’t be part of the actual test exe­cu­tion stage. Num­bers tell the story. Guts can get messy.

Report­ing and Sharing

Maybe your results will fall in line with your ini­tial hypoth­e­sis, maybe they won’t—or maybe you’ll be com­pletely blown away, for bet­ter or for worse. No mat­ter the results, though, report­ing and shar­ing met­rics and analy­sis are fun­da­men­tal steps in the test­ing process and help lay the ground­work for the repeat and improve cycle as it pulls more deci­sion mak­ers and stake­hold­ers into the process and helps make the case for more resources and man hours in the future.


So it’s over (for now)—and you’re on to the fun part. Social­iza­tion is about get­ting every­one as amped and excited as you are about the test­ing pos­si­bil­i­ties. It’s about cel­e­brat­ing your suc­cesses whether they’re true achieve­ments or sim­ply new­found knowl­edge, and build­ing even greater buzz among those around you. Be infec­tious. Suc­cess breeds success—and excite­ment and enthu­si­asm breed fer­vor, sup­port, and a deep com­mit­ment to upcom­ing ini­tia­tives. Social­ize, social­ize, and social­ize some more.

Lead­ing orga­ni­za­tions don’t stop here, though. For them, test­ing is an ongo­ing process dri­ven and refined by real-time suc­cesses and less-than-successes. This not only helps a busi­ness home in on the crit­i­cal infor­ma­tion and iter­a­tive ele­ments of test­ing, but also demon­strates sig­nif­i­cant value to inter­nal deci­sion mak­ers and fund sourc­ing. In other words, it appeals to the guy with the check­book and the “approve” stamp. If you can show true fis­cal value, look for­ward to more test­ing resources, sup­port, and in turn, big­ger and bet­ter cam­paigns and more in-depth tactics.

For more information, please visit my blog at

Discuss This Article

Comments: 0

Add a New Comment

Thank you for adding to the conversation!

Our comments are moderated. Your comment may not appear immediately.