It’s hard to open a business or a technology publication these days without finding an article about the promise of Big Data. New technologies (Hadoop, Mongo, NoSQL and others) offer businesses the opportunity to analyze very large amounts of unstructured data at a scale and speed that was simply not possible just a few years ago.
The predictions are quite bold: Gartner research recently wrote “…enterprises adopting this technology to outperform competitors by 20% in every available financial metric”.
Before you embark on a big data project it is important to separate the buzz from the reality. This post explores 5 important challenges that can limit the ability of any marketer to turn the Big Data hype into insights, business value, and ultimately, increased revenue.
What is Big Data?
Big Data can be defined a group of technologies that help organizations store and process data sets that meet one of three ‘V’s:
Recommended for YouWebcast: Build a Powerful Network and Accelerate your Growth
- Volume – for example, imagine Facebook’s database of users, posts and likes. Or a credit card company’s transaction log across all customers
- Velocity – the speed of data generation. Both examples above also require a system that processes data very fast. All this data is being created in real-time, thousands or millions of items per minute.
- Variety – what is also called ‘unstructured’ data: traditional databases operate using very well-defined schemas: name, address, credit card number, etc. Big data technologies don’t require schemas, they are more flexible in this way.
Big Data at the end of the day is a database technology. And it will not replace traditional ‘SQL’ databases, because there are things that Big Data databases cannot do, like maintaining transactional integrity, which is required for credit card payments or really any kind of business transaction.
The challenge, is that like with most technologies, the technology alone is not enough to produce business value. Speaking about this technology as a buzzword can result in inflated expectations and an unrealistic view of the steps required to take advantage of this powerful technology.
The 5 Big Problems of Big Data
It is true that data is a key resource for any marketer, or for any business person. The success stories of businesses using data to improve the business are well documented. It is also true that the promise of having infinite insights is quite attractive.
However, there could be some bumps in the journey to the promised land. Today, Big Data is more a piece of the technology than a complete solution with a clear path to business value. I can think of five main challenges marketers must sort out before embarking on such a project:
- Data sources – Most companies struggle to get good data, OK-data, and sometimes even ‘some data’ about their customers. Without an ability to extract data from your systems you won’t have much data to analyze. I have worked at organizations where customer information was scattered across 80 different databases. The dynamic nature of business today, with mergers and acquisitions, only makes the problem worse. Data integration and extraction remains a technology problem that is not solved by big data.
- Deriving Insights– Just because you have all the data in the world does not mean you can find diamonds in it. Without the right ideas, any data analysis can prove right the wrong assumption. Remember the U2 song “We thought that we had the answers, it was the questions we had wrong”. To generate insights from large amounts of data you must have the right kind of questions. If you think of Big Data as a customer crystal ball, what would you ask it? Imaging you deploy a Big Data system tomorrow that stores all your customer’s transactions for the last 25 years , now what – what do you want to know?
- Acting on Insights – Have you ever spotted a clear opportunity in your company on which you cannot act because of organizational rigidity, politics, resources or priorities? There is a Big Gap between deriving insights and acting on them. For example, most companies don’t take time to read (or act) on the customer feedback that is readily available. Many marketers don’t look at basic web analytics to act on the insights that have been at our fingertips for years. Is your organization willing and able to make changes and to act based on the insights you could generate from a Big Data initiative? Do you have a plan? Are people empowered?
- Analysis Tools – We are in the early days. Big Data requires new tools to query and report on the data. The bright future for Big Data probably will require new data integration tools, new querying tools, new reporting tools and new dashboards. I believe this one will be solved soon because of the investment being made by software companies in this space. There are a few startups with interesting technology that will get better very quickly.
- The Trap of Data – Customers are human, we make emotional decisions. Being too focused on data is a risk. If Steve Jobs acted on data alone, he would have never built the iPhone. Will you give up your instincts and gut feeling to act on data alone? As HBR notes “Human behavior is nuanced and complex, and no matter how robust it is, data can provide only part of the story. Desire and motivation are influenced by psychological, social, and cultural factors that require context and conversation in order to decode. Data …reveals what people do, but not why they do it.”. In other words, trusting data alone assumes the data you have is accurate, complete and reflects or captures all relevant aspects of consumer behavior. For example, data alone can only point to correlation, not to causation.
Does this mean businesses should ignore big data? Of course not – the technology available today brings exciting possibilities to businesses. It would be a big mistake not to try to take advantage of it, assuming you have a problem that requires Big Data and you have a plan to solve the five challenges presented above.
What I am recommending is two things: First, don’t get blinded by all the buzz of buzzwords and shiny objects, Big Data can be one of them. And second, as with any initiative, if you go at it with an awareness of the challenges ahead, you are more likely to find success. Maybe Big Success. Who knows?
If you are looking for a place to start, consider any large unstructured data source. A data set that does not require integration: the data is already in one location: transactions, web logs, multi-year sales data, customer feedback, etc. Look for a resource that has experience not only with the technology but also with the business process of extracting insights from big data and can help build an action plan around them.
However, if you are looking at big data as a way to get customer insights, there may be an easier path: one that does not require technology investments, consultants or project management. If you want to learn about your customers, pick up the phone and talk to them. No amount of computer analysis will be able to express in a spreadsheet or a report the emotions, problems and dreams in the mind of a customer that you can get in a personal conversation.
Looking at Big Data is a scientific way to get customer insights, probably sitting at the other side of the spectrum of becoming a true social business – aligning with customers and their emotions. There should be no reason why you can’t do both.