Cloud: The Impending Data Management Debt Crisis

There are a lot of great business reasons to be excited about the cloud. For a lot of organizations this means vastly increased business agility; they can stand up a new project environment in minutes or hours instead of months or quarters.

Gone are the days of buying hardware servers, operating systems, networking, storage, security, administration, etc.

The problem here is that a lot of people are assuming that a good Integration Platform as a Service (iPaaS) product is all that they will need to be successful with cloud applications and systems. iPaaS is a great place to start on this journey, but is not the complete solution.

What iPaaS provides today

Here are some of the common capabilities provided by an iPaaS product offering:

• Connectivity to data sources and targets
• Batch Data Integration – extract, transform, and movement of data
• Application integration – typically for both sending and receiving data from applications via APIs such as REST
• Process integration – the ability to sequence integration actions into a process flow

What iPaaS does not provide

iPaaS does provide flexibility to manage a number of different data integration usage patterns that are common to the cloud. What is does not provide is a comprehensive data management solution. If you are planning for your systems to be a “source of truth,” a “system of record,” or to provide data that will be critical to some analytics process you may be operationalizing, you are going to want to ensure that the data in those systems is clean, complete, timely and secure. So what is missing? Here are a few examples:

• Data Quality measurement and implementation capability
• Data enhancement or enrichment capability
• Master Data Management capability – to master and relate all of the data pertaining to a particular subject (customer, product, partner, etc.)
• Data Security – to ensure that sensitive data is protected as is it is moved around both inside and outside of the organization.
• Real Time data delivery – increasingly, decision making and customer interaction systems depend on trustworthy that is delivered in real time.

Over time, we expect iPaaS to start to fill some of these gaps, but in the meantime, they are not being addressed.

A Cold Dose of Reality

With this understanding that iPaaS will not address all of your data integration challenges, where should you start?

First, if you are going live with a stand-alone instance of Salesforce.com or something similar, then iPaaS may meet your needs, for the present. The problem is that no organization of significant size can afford “stand-alone” applications. This is the age of digital transformation. Data is the fuel that organizations will build their competitive advantage on. Nobody can afford to continue to have silos of data that are unconnected to the rest of the systems in the organization if they are to deliver better customer experiences, better patient healthcare outcomes, faster and better management decisions, etc.

Second, if you are moving data around without ensuring that it is accurate and trustworthy for use in analytics and decision-making, you are taking a big risk. Executives at the majority of large organizations say that they simply do not trust the data behind their reports and dashboards for important decisions. Analytics for important business decisions and processes must be based on data with actively managed quality, completeness and timeliness.

Third, the fact that all of your data is connected through some form of data integration is a good start, but to really make use of it you need to determine how the all data relating to entities such as a customer, for example, relate to each other. This is how you will get a full, intelligible, 360 degree view of your customer rather than a collection of raw data.

And finally, if you are moving and sharing sensitive data that is unsecured, you are taking a big risk. Creating new risk can be as simple as copying data to a new development environment, or combining data in a data lake in ways that turn non-sensitive data into sensitive data. We have all seen the headlines that show the results breeches and data loss.

Cloud does not mean that the data management fundamentals have changed

Grim as this may sound, there is really nothing new here. What has been important for data management on-premise is also important in the cloud. Some organizations are operating as if the old best practices are unimportant or they are waiting for some unknown, new data management technology to spring up and magically solve the problem for cloud users.

Organizations that defer good data management practices are in reality running up a technology debt. The data management issues that they do not address now, means that the data problems will get worse over time (think of accruing interest charges) and it will cost much more to fix later. Creating a new cloud system with minimal data management is going to create new “silos of data” that make it hard-to-impossible for other new systems and analyses to discover, access, and manage the data for new purposes. And the greater the number of data sources and their use, the greater the interest multiplier on the technical debt.

“Kicking the can down the road” approaches to data management may enable organizations to meet deadlines, but they need to understand that they will leave these organizations poorly prepared for the digital transformation that is sweeping through virtually every industry.

Requirements of a data management platform for digital transformation

One of the core problems here is that IT is struggling to meet the data requirements of the business side of the house. What is called for is greater agility and more self-service without compromising quality, security, and timeliness of the data. Look for a data management platform that:

• Provides an end-to-end solution across cloud, on-premise, big data, and hybrid
• Can manage any type of data and any business use case
• Provides a high level of automation to maximize IT productivity
• Enables business self-service, that without compromising data quality or security
• Accelerates the work of both business and IT though the use of metadata to fuel intelligence in terms of recommendations and suggestions
• Enables business and IT to collaborate at a more meaningful level to enable next-generation data governance and security

It will be a challenge, but if you start now, you can lay the data management foundation to support your organization’s digital strategy.