The ABCs of A/B Testing: How to Base Your Subject Lines on Science

When time-starved customers are bombarded with emails every day, your subject lines can make an enormous difference in whether subscribers actually see your messages.

In fact, recent research shows 47 percent of email recipients quickly decide whether to open or ignore emails based on the subject line alone. Worse, 69 percent of readers who report email as spam are motivated to do so by how the subject line reads.

Of course, psychology plays a major role in that behavior, and researchers are continually trying to determine the parameters involved. For example, Experian last year found personalizing subject lines led to a 42 percent lift in unique open rates for the consumer products and services segment, an improvement that averaged out to 29 percent when applied across industries. However, personalization was completely ineffective in emails pitched to publishers, and it only improved opens by 1 percent when sent to media and entertainment representatives.

Because applying reader psychology to the email creation process is an inexact science, the only way to determine the most effective wording of your subject lines is often via A/B testing.

“There are limitless possible emotional combinations of language available,” observes Parry Malm on Econsultancy.com. “Knowing which work best for your audience is impossible without testing. The opportunity is to quantify emotion … then optimize based on the results … then profit from better subject lines.”

Fortunately, such testing is easy and convenient through the use of online tools.

VerticalResponse, for example, boasts an A/B testing feature that allows Pro Plan users to quickly and easily compare alternate email subject lines. Two different subject lines are sent to 25 percent of an address list, and within minutes of initiating the tests, the technology automatically sends the subject line that drew more opens to the remaining contacts, boosting the open rates for the entire campaign.

Testing options offer choices for better open rates

So how does A/B testing work? As its name implies, it helps you make decisions in your email campaign by temporarily presenting two different options to your audience, then gauging response rates to determine which option is better received. Here’s what the process generally entails:

1. Determine which factors you’d like to test. Before getting started, you need a good handle on the subject-line variables you’d like to compare. Industry research in that regard is valuable, but you often can’t determine what will work best for your specific audiences and circumstances without going directly to the source.To think about elements to compare, consider the questions that have occurred to you as you’ve tried to finesse your subject lines in the past. For example, you may be wondering:

• Are short or long subject lines more appealing?
• How would my audience respond to numbers and dollar figures?
• How would recipients react to their names in the subject line?
• How much other customization is advisable?
• Should my subject line be phrased as a statement or question?
• Are exclamation points OK?
• Should I use industry jargon or more colloquial terms?
• Will humor be well received, or should I maintain a straightforward tone?Once you’ve established your most burning questions, you can create a spreadsheet allowing you to record and compare results.

2. Establish logistics. In designing your test structure, you must establish ground rules so results aren’t skewed; that way, you gain real, scientific insights. Aim to control as many extenuating variables as possible. For example:

• Your test groups should be equal in number.
• Your tests should run for an equal time interval.
• The tests should go out at the same time of day.Run your tests as frequently as you’d like. After the completion of each, you can fill in the results on your spreadsheet, after which you can take them up for consideration when crafting future emails.

Some common variables

As mentioned, you’ll need to decide which variables are most relevant to your campaign. However, you may want to be aware of research already conducted and how that might affect your own testing. You might also consider the following best practices in getting started:

• Question versus statement
  Sources differ on the advisability of incorporating question marks into your subject lines, but many recommend against it on the grounds the recipient may not open the email if his immediate mental answer to your question is negative. One source recommends instead making the subject line a statement, like “Question about your career,” which could prompt the reader to click to learn the nature of the question.
• Short versus long
  Brevity is of the essence when trying to appeal to busy consumers in a world vying for their attention. One study found subject lines of six to 10 words gleaned the most opens (21 percent), followed by those with zero to five words (16 percent) and 11 to 15 words (14 percent). Note 68 percent of emails are opened on mobile, and most mobile devices display only four to seven words of subject lines.
• Humorous versus straightforward
  That depends heavily on your target audience and whether it veers toward the conservative. Obviously you should stay away from anything that’s offensive, but in general, audiences respond positively to clever lines that are unexpected or make them smile. One study also points to much higher open rates for subject lines readers view as original. Still, overly ambiguous lines may lose you fans; one source points to 56 percent better open rates for messages that are very clear as to their intent.
• Emojis versus no emojis
  Salesforce reported last year on the increased use of characters and emojis in subject lines, estimating some 2 percent of B2C subject lines included them last year. “Using special characters and emojis in subject lines may seem gimmicky or silly,” advises Chad White in the report. “However, adding these visual elements to subject lines has proven quite effective for some brands, which should come as no surprise since adding images to tweets, blog posts or just about any other media also improves response.”
• Personalization versus no personalization
  Research differs on the extent to which personalization improves open rates. But most analysts believe it’s a good concept to embrace as consumers increasingly expect big data to help customize their business transactions. “Consumers expect their interactions with a brand to be dynamic and personally relevant,” notes Shelley Kessler on MarketingSherpa.com. “A brand that recognizes a subscriber by name in the subject line of an email is able to establish an immediate, personal connection with that subscriber, and that is reflected in our research.”

Recommended reading: Our Vice President of Product, Thomas Fanelli, gives readers five ways to to get started with personalization.

Also worth your consideration

Recent research points to the following other helpful information as you attempt to create meaningful subject lines for your emails. However, we still recommend testing these concepts on your own to determine whether they hold true for your audiences and circumstances.

• Email opens decrease 19 percent when the word “newsletter” is included in the subject line.
• Emails with no subject line are opened 8 percent more often than those with a subject line.
• Emails with “FW:” (denoting forwarding) in the subject line are opened 17 percent more often.
• Text that creates a sense of urgency and exclusivity improves open rates 22 percent.
• Using the words “daily” or “weekly” in subject lines boosts open rates, whereas the word “monthly” is detrimental.
• Subject lines that simply describe email content are more effective than those including a hard sell or call to action.
• Subject lines incorporating the word “free” are opened 10 percent more often.
• Other words that often perform well in subject lines include “content,” “video,” and “news.” Words that typically do more harm than good include “perfect,” “good,” “donate,” and “report.”
• Certain words in subject lines are more likely than others to derail your email to spam; learn some of them here.
• Subject lines incorporating passive words perform up to 14 percent better than those with active words.
• Fewer than 1 percent of B2C subject lines include hashtags, which can deter opens but are nevertheless useful in pointing traffic toward social media campaigns. “Hashtags let you tap into big ideas and themes,” notes Chad White at Salesforce.

Recommended reading: See more magic marketing words you should be using.

Ready to get started? A/B testing requires some thought, but it can help narrow down your email marketing strategy so you can take maximum advantage of one of the least expensive and most effective marketing tools available.