We’re entering an era where organizations are looking to move beyond the traditional data warehouse. But the data warehouse is not going away just yet. Far from it. Data warehouses will remain an indispensable part of the analytics toolbox for a long time to come—but they won’t stand alone.

When I talk to customers, most tell me they’re planning to modernize their analytics in terms of:

  • Augmenting the data warehouse with cloud data warehouses such as Amazon Redshift, Microsoft Azure SQ Data Warehouse, Snowflake and others.
  • Augmenting the data warehouse with data lakes and business self-service.
  • Augmenting the data warehouse with NoSQL, columnar and other types of specialized stores.
  • Upgrading to real time and streaming.
  • Adding predictive analytics, machine learning, deep learning, etc.

These multiple conversations tell me that there’s a trend. But digging a little deeper, I see three really interesting trends.

First, as applications and business processes become more standardized, organizations are embracing the idea that the best source of competitive advantage is in the use of data, internally and externally sourced. This means analytics-driven insights can drive better, faster decisions, and enable better business processes and customer interactions. But the difficult challenge is that IT organizations are being asked to deliver trusted and secure data from a much wider variety of sources much faster than ever before.

Secondly, after years of consolidating enterprise applications into integrated ERP systems, the pendulum is swinging the other way. Now new cloud applications—Salesforce, Marketo, Eloqua, Workday and NetSuite—are breaking up the integrated application stacks and spreading them to the cloud. A primary reason is that these applications are able to evolve and adapt much faster to meet the needs of their business users.

Third, that explosion in new analytics technologies includes the particular growth of cloud and big data for analytics. With all of this, there are tremendous challenges for CIOs, chief data officers, and enterprise architects.

The IT Leader Challenge

Just when the business is requiring that IT deliver trusted data faster than ever, these trends are conspiring to make it that much more difficult for IT. And you have to consider that IT budgets have been pretty much flat worldwide since 2008. Clearly, the only way for IT to keep up is through more standardization and automation of the data management processes—across the entire enterprise-wide data spectrum. The interesting thing is that a TDWI survey has shown that an integrated data integration tool set is the No. 1 desired change among the managers they surveyed.

The Architect challenge

There can’t be much debate about the need for data management standardization and automation within the architect community. The problem is how to accomplish this without slowing down the current business. The standard approach is to start small, show value, expand gradually. This can work if you are building toward a clearly defined future-state architecture. Otherwise, you could produce a series of one-offs that not only fail to advance the architecture, but ultimately hobble it.

The Chief Data Officer Challenge

The role of the CDO is to drive the use of data and analytics to provide a competitive advantage for the organization—and increase revenue and profit (IDC pegs the value of productivity gained by using all data in analytics by 2020 at $430 billion[1]). The CDO will be critical to the success of the organization specifically as the person who sets the priorities. Not all data can or should be managed. Not all business initiatives are the top priority. It is important for the CDO to set priorities on the business initiatives, and then to make sure the owners of each business strategy also own the data that will drive their success.

Everybody’s Challenge: Our reach exceeds our grasp

The challenge for everybody is that while BI/Analytics remains the No. 1 spending priority for CIOs for the fifth straight year, a recent McKinsey and Co. study found that 86 percent of companies surveyed say that they are at best only somewhat effective at meeting the goals of their data and analytics initiatives. The CDOs, architects, and IT managers need to collaborate to deliver on the business potential of all the data every organization is literally swimming in.