It’s been a funny old year for research. Right up to the day before the UK’s June referendum on EU membership, polls were predicting a narrow margin of victory for the Remain campaign. Many added that wavering voters would beat a last-minute retreat to the relative comfort zone of the status quo. As it turned out, of course, there was a narrow margin of victory for dynamiting the status quo. Five months later, also contradicting the pollsters, a narrow (and at the time of writing, disputed) US presidential election result delivered another raised middle finger to business-as-usual.

Missing emotions

Writing for the respected UK research website Election Data, Ian Warren bluntly articulated the question on many people’s minds: “WTF just happened?” Warren used this profanity to illustrate the powerful role that emotion played in swinging both these results. “Vote leave, take control” and “Make America great again” (however they might ultimately play out) were powerful, emotional calls to action in a way that “Stronger In” and “Stronger together” were not. And as we now know, voters made their choices in the face of a barrage of objective, factual information strongly suggesting their decision could have destructive consequences. This is the world of 2016’s Word of the Year “post truth”, defined as “relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief”.

Warren says that the pollsters failed because they missed out this emotional factor. Pointing out that the Leave vote was high in areas of low immigration, he notes it was perception, not reality that mattered. The problem with the pollsters’ models, he suggested, is that none “seem to incorporate the most important factor in people’s voting decisions: what they think and how they feel. It’s the question which the data rarely gets to.”

It’s as if all the research into voting intentions asks just the rational, “left brain” questions while ignoring the emotional “right brain” issues.

Mostly about feelings

Businesses do this a lot. We regularly find a mismatch between what people want and what’s being pitched to them. As the managing director of noise monitoring specialist Sonitus Systems realised: “You need to figure out why the customer would buy it from you, not why you think the customer should buy it from you. And if you don’t work that way, somebody else will.” Once this electronics company got the bottom of what was motivating its customers, it was easy to tweak its products and messages to get an immediate improvement in sales.

In another example, a large financial services company had run traditional qualitative and quantitative research for five consecutive years in an attempt to find out why customers were leaving it. None of the changes it made based on the “rational” findings of these surveys made any appreciable difference. Only when the company used psychological techniques to interview and observe customers did it uncover the thoughts and feelings behind their disloyalty. It turned out that customers were annoyed – sometimes even actively hostile to the brand, for complex emotional reasons. They felt they were not being heard and they didn’t want what they were being offered. Like the voters, they wanted change. Using this insight, the company was able to put together a value proposition that took these feelings into account. Within just six months, transactions, renewals and new business enquiries were all trending up. As with the UK and US elections, the facts (in this case, rational product benefits) didn’t matter to these clients. It was mostly about feelings.

Behavioural subtleties

Today, we risk losing touch with this valuable source of insight unless we keep up human contact. Writing in Nautilus, Stanford professor Robert Sapolsky points out that for 99% of our history, social communication was face-to-face, giving each person access to the full richness of the other’s being. He notes that you know someone by “their microexpressions, their pheromones, their totality” and that since the advent of letter-writing, technology has progressively mediated this direct experience, causing a “withering of primate familiarity”. While social media in particular promises always-on intimacy with a wide range of people, it is a false familiarity. It misses out on the many nuances, intonations and non-verbal cues because these are masked or obliterated online.

Quantitative research, whether into voting intentions or customer behaviour, can show us the trends and the direction of travel. But it can’t tell us why people do what they do. People rarely express the feelings behind their intentions over the phone, online, or via social channels. So unless they are skilled in what to look and listen for, researchers miss the behavioural subtleties.

How to bridge the gulf of understanding

And even face-to-face interviewing can miss these insights if it doesn’t use the right techniques. This is because people behave differently in the presence of a professional researcher. Getting the real, rich, heartfelt feelings that can make all the difference in business means forgetting about being an “objective” interviewer and experiencing a situation in the same way that the other individual experiences it. Psychological techniques like IPA can uncover these insights. When we’ve used them, our commercial clients are always astounded by the richness and depth of insight they reveal. At last, they find they have a foundation for positive change.

We’ve heard a lot in recent weeks about how people feel distanced from political decision-makers who neither listen to nor understand their concerns. This mirrors what happens in business every day, as companies try and bridge a gulf of understanding with their customers using arms-length, mechanistic research techniques.

If they want to avoid more “WTF” moments in the future, politicians and business leaders alike need to get closer to their constituents and do a lot of deep watching and listening.