In my previous post, I talked about the idea of interwoven data stories and mentioned the difficulties companies encounter in managing, integrating, and gaining insights from data. In this post, I will explore data fabrics and how they are becoming a simpler way to unite data.
For the past few decades, combining data usually meant setting up a data warehouse and moving data into it. This has never been an easy job. Gathering data from different sources required designing the warehouse (like adding shelf space) and extracting (moving data from its source), transforming (adjusting data to fit those shelves), and loading data (putting it on the shelves) in the warehouse.
Figure 1: Conventional data integration
While this conventional thinking was effective and useful for years, things have changed. Big data happened. Data has gotten bigger and more complex, and organizations are scrambling to get their arms around it all. Big data impacted conventional thinking in three ways:
Data sources became vastly numerous. Conventional thinking about data integration began to breakdown as companies begun to amass a number of individual heterogeneous data sources in the thousands. These sources included core systems like CRM and ERP, but also those rogue Excel spreadsheets and other data stores that seem to pop up every other day. Many companies have added to that volume the complexity of dealing with and managing cloud-based and Software-as-a-Service (SaaS) systems.
The types of data became immensely varied. Data is no longer just the run-of-the-mill variety, such as customer addresses and sales transactions. Sensors, embedded medical devices, GPS, server logs, website clicks, documents, crowd sentiments, social media, video, audio, photographs and the emerging Internet of Things have entered the data fray—making conventional data integration even more complex.
The velocity of incoming data accelerated rapidly. Data trickles became fire hoses as the volume of incoming data has ballooned. Conventional data integration is often too lengthy a process to keep up.
Figure 2: Big data happened
The result of these massive changes has some industry analysts and researchers estimating that only 1% of the data a company has actually gets integrated into a centralized data warehouse. The reality is data sources are too numerable, data types are too varied and data is coming in too rapidly for any organization to fully integrate and manage it all and holistically leverage it for insight. It is simply too much with which to keep up.
Thankfully, an alternative to conventional data integration has emerged—the interwoven Data Fabric. Think about the term “fabric.” Say an organization was to draw a diagram of every single data source they have on a piece of paper. Then they drew lines to connect all those sources the way they would like them to be connected or integrated together. The number of lines would be so numerous and weave back and forth so densely, it would look like threads of a woven piece of cloth—a fabric, if you will.
This is where it gets interesting. What if this fabric itself was a data source—the data source for all your analytics? What if all the data sources in your company’s data ecosystem were instantaneously connected and interwoven at once?
Figure 3: An interwoven data fabric
The data fabric allows users to view all data sources as if they were a single database. This simplifies data access considerably by creating a single point of entry to all the data in the ecosystem. Simply point your favorite BI tool (or even Excel) at the fabric, and all the interconnected data is available at your fingertips!
A data fabric also works like sheets on a bed, so that all the orchestration, query execution, syntax and processing happens under the covers. For example, data agents within the fabric strategically plan and manage data movement to minimize network traffic and processing time—all behind the scenes. This way the business user does not need to worry about the technological whirring under the hood. The data fabric separates the accessing of data from the processing of data, making it easier for companies to bring together existing data sources and add new ones—Hadoop, Cloud, you name it. For many organizations, a data fabric is the solution they have been needing to breakdown data silos and provide holistic business insights in a complex and modern data environment.
If you had a fully interwoven data fabric within your organization, what insights could you gain? What competitive advantage would that mean? Consider a data fabric to move beyond conventional thinking and put holistic business analytics on the fast track.
Join me and learn more about interwoven data quality with the free webinar on 12/16 at 2pm ET.