Everybody knows that big data is a driving force in business and it’s more important than ever to collect and understand information about your customers, competitors, and business process. The problem with big data is that drawing valuable insights from the mountains of data produced by your business, as well as external data, defies efforts to collect, organize, analyze data leading to valuable insights that optimize your business for success. Instead, businesses flounder with too many reports and too little insight. In his book, Web Analytics, 2.0, Avinash Kaushik (of Google Analytics fame) begins with a story of deleting 200+ reports soon after joining an early employer only to find no one even missed the reports enough to ask where they’d gone because the reports didn’t contribute understanding or insights that made a difference in decision-making or planning. The trick with big data is untangling the mass of data that looks like tangled Christmas lights. to derive valuable insights, a problem depicted in the image below.

valuable insights

Image courtesy of toward data science

The tools for collecting data are better than ever, leading businesses to believe they should capture every bit of data available. But what can you do with all of that data? The answer is that many businesses, like Kaushik’s first employer, do nothing of any value with the data.

So many businesses, especially small companies, find themselves buried underneath a huge mountain of data and they have no idea how to manage it. They’re not drawing any useful insights from the data they collect, which means that collecting and storing data, with the associated costs, is a waste of time. If you feel daunted by the idea of big data management, here are a few ways to help you sort through the pile and find the important insights.

Derive valuable insights from your data

First, a word about your data.

Data realities

According to IBM, 80% or more of all the data in the world is unstructured, such as words, video, and images. Unfortunately, we don’t have great tools for analyzing unstructured data and rudimentary tools such as NLP (natural language processing) suffer from the nature of language, which lacks specificity. Think about your native tongue and you realize very quickly how interpreting meaning involves more than the words themselves, including intonation, body language, context, and even the relationship between those involved in the communication. That’s why texting is a poor substitute for complex communication.

Even if we only consider the 20% of the information out there that’s numeric, big data represents a serious challenge; 4 in fact.

  • Volume – current estimates place the volume of data at 40 zettabytes (300X more than in 2005), a number increasing by 2.5 quintillion bytes per day.
  • Velocity – data comes at you at an increasingly fast rate. In fact, by 2025, experts expect that 30% of data collected involves real-time data (check out the graph below)

real-time data

Image courtesy of ZDNet

  • Veracity – is the data collected by your organization accurate? For instance, I worked on a project for a client involving Google Ads and Google Analytics where each platform reported a different number of visitors from advertising. That shouldn’t happen.
  • Variance – firms face data coming in many formats and from many sources. Bringing all the data together to drive valuable insights is challenging, involving using SQL and Python to construct databases comprising data from different systems, such as your sales software, CRM system, and inventory management. Remembering the huge problem with unstructured data, variance also refers to making interpretations based on this qualitative data.

Analytics challenges

Even once you have all your data in one place, you pre-processed it to get it in a usable form, and cleaned the data, you must analyze the data, which requires a unique combination of data analysis skills and business acumen. Don’t believe me? Knowing the right questions to ask of your data; questions yielding valuable insights to drive better decision-making requires business acumen, and actually doing the analysis requires skills in data analytics.

Finding folks with this combination of skills isn’t easy or cheap. In fact, if you want a career with a ton of upward mobility, lots of employment options, and a high salary, business analytics is the right choice.

How to manage your big data challenge?

Rethink what you collect

There are people out there who advise you to collect any and all information you can about your customer because the more data you have, the better you know the customer, which translates into improved marketing campaigns and higher sales. The thing is, if you are overwhelmed with data to the point that you can’t do anything with it, that strategy doesn’t really work. It’s actually better to streamline the data that you collect and focus on the important bits that directly lead to valuable insights.

If you can cut back on the amount of data you have in the first place, it’s a lot less daunting to manage. Plus, storing data is expensive. Cloud storage costs somewhere between $.02 and $.026 per gigabyte per month, although costs change when you have more than 50 terabytes per month. And, those costs add up as you store more and more data every month.

Use multiple collection methods

Collecting data from a wide range of sources is the best way to build a full picture of your customers. If you only focus on data gathered from your website, for example, you’re missing out on a number of potential customers you interact with on social media and you don’t have the robust information about actual customers available in your CRM and invoicing software.

Use as many different data collection methods as possible to provide a deeper pool of information to draw insights about customers and prospects. However, use discretion to only collect information necessary to create valuable insights.

Using a key associates related pieces of data from different databases to allow foster insights. Often companies use a customer’s phone number of another unique key for this purpose. Then, it’s a simple task for SQL to draw insights across the databases.

Implement ECM software

Finding ways to organize and manage all of the data that you collect is so important if you want to make sense of it all. That’s why digital transformation tools like ECM (enterprise content management) software make a great addition to your toolbox. If you don’t have any form of organization for all of your business documents, especially those that weren’t digitized when created, you end up with a chaotic mess of data that you can’t do anything with. Once things are organized and you have a clear picture of what data you actually have, it’s much easier to start analyzing it.

Have a clear goal in mind

Not having a goal in mind is one of the biggest mistakes people make with their data.

  • What are the questions you have about your customers?
  • Do you want to build a general profile or are you looking for specific information about their buying habits?
  • Is there a particular demographic or other segmentation information you want to focus on? For instance, are the buying habits and preferred products different across buyer segments?

Building valuable insights relies heavily on guided curiosity about your customers and how they interact with you.

Final thoughts

Wading through all the data produced by even a moderately sized business is challenging, but if you follow these steps, finding valuable insights is easier.

If you have ideas for future posts on this site or questions about this post, feel free to enter them in the comments below. I’d also like to hear how you glean insights from data in your organization.

Read more: Insight vs Analysis vs Reporting