When marketers think of using data to come up with test hypotheses, they typically think of digital analytics. Quantifiable data. But qualitative research can actually offer more insight than anything else for coming up with winning test hypotheses. When quantitative stuff tells you what, where and how much, then qualitative tells you ‘why’. The goal of qualitative research is to gather an in-depth understanding of user behavior, and the reasons for that behavior.
You have a 4-step sales funnel on your website. Most people have no problems taking the first step, but almost all of them drop out at step #2. Why? This is where analytics data can fall short, and won’t be enough to figure it out. You need to know what your actual users are thinking when they see that page, and leave. And the only way to get that information is through asking your users.
Qualitative research enables us to peek inside the mind of the people we’re trying to sell to. Without knowing what they’re thinking, we’re in the dark. We can speculate all we want, but it’s not going to get us anywhere.
How you conduct qualitative research matters
Qualitative research involves people. What you ask them, when you ask them and how you ask them will all influence their response. They might lie to please you, they might be influenced by how you frame a question (“Don’t you think this looks more secure?”).
Michael Summers, Global User Experience Research team at PayPal:
There are two very precious cognitive resources in the mind that influence how we study optimization:
- and external motivation.
We all have a sincere desire to understand how our UI is supporting conversion — or what obstacles or barriers exist. However if you imagine a young child with a large magnifying glass — who leans over fragile flower to study it more closely, only to accidentally concentrate the suns rays and cause the flower to burn — sometimes when we try and “study” something — the actions we take to understand it get in the way.
When we try and watch people encounter our UI or our conversion funnel, perhaps arriving at a landing page, evaluating options, considering our “pitch,” or value proposition, and then perhaps sign up — if we artificially distort attention and motivation, we will walk away with unrealistic values for things like comprehension, retention, decision to act, time on task, success rate, and error rate.
For this reason, the way we study people matters. A lot.
How do top experts use qualitative research to get insights?
I reached out to fellow conversion optimization people, and asked them how they are going about qualitative research. Which research methods have been most useful to them, and how what specifically have they asked the users, or have them do to get insights?
User testing and usability testing are the favorites for many optimizers.
Dr. Karl Blanks from Conversion Rate Experts praises user testing:
User tests allow us to see how visitors interact with a site, but they also have other benefits. Each user test gives us a deeper intuitive understanding of how people use websites. And they allow us to collaborate with the user, so we can work out solutions to difficult problems. No other technique delivers so many “insights-per-minute” on a prolonged basis.
Craig Sullivan, optimization guru:
By far the most useful technique is user testing, with as early stage of a product or new feature as you can get. This is where the best blinding insights, product pivots, winning tests or great ideas came from in my work.
Start on paper or using some of the great prototyping tools out there – and get feedback, early and often as you iterate and build something out. I must also wave the flag for Voice of Customer or Feedback tools, particularly where you are measuring customer satisfaction (e.g. NPS) and service ratings for multiple elements of the product. Working out where to invest for increasing delight works perfectly with an optimisation programme.
Moderated user research
User testing has to be handled with care – and often moderation is needed to make sure we’re not just having the testers go through the motions.
Over the last 15 years of conducting user research, I’ve witnessed a range of different approaches and moderation styles, but unless you’re careful you can invalidate the research. It’s crucial when using this research technique to do the right preparation, the right recruitment and use the right style of moderation. Follow this and you can’t fail to gain powerful and highly actionable user behaviour insights, which to date, have led us to developing some of our most impactful A/B test hypotheses.
Moderated research comes under criticisms such as “people will tell you what you want to hear” or “this isn’t an environment where people usually browse”. These can be negated with the right facilities and welcome questions prior to conducting any research scenarios, allowing participants to provide their thoughts in a neutral setting. Plus get the right level of observation, probing and non-leading questioning and people have a phenomenal ability to unlock and share the hidden motivators in their brain.
A quick tip that we often use to gain more considered feedback and insights from people – provide at least two different digital experiences to compare to each other. This allows people to progress their thinking to what really does matter for them, and helps to identify strengths and weaknesses across the different experiences.
One final often overlooked opportunity using moderated research, is how it can be used to gain insight or validate the fundamentals of your businesses. Conducting research on your proposition to see if it resonates and persuades your target audience can provide you with strategic insights and the opportunity to gain advantage over your competition.
Michael Summers, Global User Experience Research team at PayPal chimes in again:
The best way to learn about your UI, is to mix it in with competitors, or even non-related tasks, and ask people to conduct a very general goal (“you are considering finding a new insurance company, please click on these bookmarks and determine if any of these offers are a good fit for you”).
By doing rapid on-line tasks in sequence, and not even revealing to participants what you are studying, you are less likely to artificially slow them down, and also less likely to impose external motivation such that they understand more than they would under natural conditions.
Also, before you even do your blind set of tasks — ask users to spend some time on-line without you directing them at all. Allowing them to read the news, check Facebook, etc., will help them acclimatize to an unfamiliar environment and begin to move at a more realistic tempo to what they would do at home on their couch.
Live chat transcripts
Do you use live chat on your website? It’s a proven tactic for increasing conversions. You can also use it to get insights.
Live Chat transcripts tell us what people are looking for who don’t want to search. It also tells us what is missing from a site or what is hard to find.
Sometimes you can feed the transcript into Wordle.net to tease out words that are used frequently, giving you clues to larger issues.
Interviewing customers, sales and support people
Interviewing customers and stakeholders is without comparison the most insightful qualitative research method in my CRO toolkit. In my experience nothing beats actually talking to your target audience. The insight you get is priceless, and no amount of quantitative data will let you reach the same level of understanding.
When it comes to copywriting and messaging, I find customer interviews to be especially insightful. In many cases there’s a huge difference between what marketers think the target audience needs to hear – and what the target audience actually needs to hear in order to make a positive buying decision.
I’ve been involved in several optimization projects where customer interviews revealed that the core value proposition was fundamentally flawed. Moreover, the answers I got from these interviews got me much closer to the winning optimization hypothesis.
I also find it extremely useful to conduct interviews with employees from Customer Service, Support and Sales. These guys and gals spend all day talking to customers and have in-depth knowledge of the problems and issues that they are dealing with – both in relation to the website and the product itself. Moreover, they are familiar with the decision-making process of the target audience and can help us build better optimization hypotheses.
When I interview employees from Sales and Customer Service, one of the most useful questions is very, very simple: “What are the five most common questions you get from (potential) customers?”
Followed up by this question: “What do you answer when you get these questions?”
Try it out next time you are conducting qualitative research for a CRO project. I bet you’ll be amazed at how much insight you get.
One of my own personal favorites is customer surveys. Surveys are mostly about learning who the customers really are, what they want, what’s causing doubts and hesitations, and the language they use. This is critical for copywriting, understanding friction , learning what matters to them about the products you sell, and so on.
You want to survey only recent first-time buyers. People who don’t have a history with you, no prior relationship. And no more than 30 days since their purchase (better if it’s right after completing the purchase) – or they will forget what they were thinking when they shopped on your site (and then bullshit their answers). The best online surveys have open-ended questions. No (or very limited) multiple choice questions.
What to ask? It depends on the business / website. Broadly speaking, you want to understand 4 key things:
- who they are – pay attention to how they self-identify, useful for putting together personas;
- user intent – what’s the specific problem they were solving;
- shopping process – what mattered to them when choosing the product, what kind of comparisons did they do, how many / which other sites they looked at, and so on;
- friction – fears, doubts and hesitations they experienced before making the purchase.
Dr. Karl Blanks from Conversion Rate Experts:
We survey people who have just signed up or purchased something. The golden questions for them are these:
- What’s the one thing that nearly stopped you buying from us?
- What was your biggest fear or concern about using us?
Web traffic surveys
Most people on your site will not buy anything. How can we get more people to buy? One thing that helps us figure that out is website surveys.
There are 2 ways to survey your web traffic:
- Exit surveys: hit them with a popup when they’re about to leave your site.
- On-page surveys: ask them to fill out a survey as they’re on a specific page
Both are useful. What to ask? Since our goal is to get more people to take action, start with learning about friction. What are the FUDs (fears, doubts, hestitations) they are experiencing – while on a specific page?
Step 1: Determine the most wanted action for the page.
Step 2: Come up with a question that asks about friction.
So for instance if this is an ecommerce product page, the goal would be cart adds. So the question to ask could be something like “What’s holding you back from adding this product to the cart right now?” or “What’s keeping you from buying this product right now?”.
See if you spot any trends in the responses. See if your customer survey responses about friction are similar to web traffic survey results. You’re mining for insights! See if you can validate or invalidate some of your observations from the heuristic analysis.
A bunch of people will tell you that pricing is an issue (“too expensive!”) – that’s to be expected. If that’s a dominant response, you’re either driving too much unqualified traffic to the site, or you’re not doing a good enough job communicating the value of your product(s).
Qualitative research helps you look inside the mind of your customers. You will learn what they really think, what they want and how they want it. If you can’t figure out WHY people are dropping off here or there, or how to increase motivation to take action, qualitative research can help.