Data is like water. It flows from one place to another. Big Data on the other hand is like the ocean, and you can very easily drown in it. It’s another reason why the ‘relevance paradox‘ will be the make or break of business decisions made from the informational tidal wave.
A quick Twitter exchange with Ray Wang from Constellation on information brokering lead me to the idea of data plumbing rather than what some analysts are calling for which is information brokering and data scientists. Ray posited that with big data you’ll be able to subscribe to streams like cable channels, and in a way this is already happening in your every day life: you follow people on Twitter and subscribe to their info-stream, you connect with people on Facebook and LinkedIn and do the same. However too many connections can lead to data overload, or in some cases just a load of garbage that you don’t want to read. But you want to remain connected with these people because now and again they do say something relevant.
So why a plumber, why not a scientist or even a cable engineer ?
Because data is alive; it grows, ebbs and flows as it moves and collates information. Data is in fact non-linear despite the pictures painted of information highways, it doesn’t get into it’s battered Ford Taurus and drives from A to B, it’s very much like a stream meandering from one point to the next, collecting stuff along the way, carving out new channels.
Scientists do the number crunching and make sense of the flood of information, but it’s the handy plumber that makes sure you get the information you need, when you need it, and by how much you need, routing the flow of a constant stream of acquired knowledge and making sure there are no leaks. You want to understand or make a decision on something you turn on the tap and the information flows to you, instantly. It’s not there when you don’t need it. It’s not constantly dripping and distracting you.
The scientist maintains the water tank, the plumber maintains the system itself.
Who do you entrust your data to more ?
Good post. With a focus on collecting more data or even on analyzing it, marketers will suffer from analysis paralysis. But I’d take it a step further and say that for some applications, the marketer needs to be cut out of the big data collection->analysis->action flow.
For companies that must act on consumer demand in real-time, they need a “big data app” as the plumbing that interprets consumer intent (what do they want) and delights them by serving it up immediately.
A very good analogy but one that a lot of organisations have trouble accepting. The big challenge is to get people to move away from the idea of having one person/team/department in charge of the whole “big data” life cycle and introducing separation between collection, regulation and analysis.
Justin’s comment above about a “big data app” to provide the plumbing makes me think of the likes of BrightContext, exactly the type of thing they’re trying to provide.
Its a nice analogy, although i’d argue that you need an effective collaboration between those two entities for a “system” of data acquisition and augmentation to succeed, all the plumping in the world is not going to help a leaky tank!
There are already third party data aggregators and suppliers like Gnip, and data scientists for hire like Opera Systems and the ranks are growing. Why would thousands of companies hire data scientists to make sense out of the flow from social and public data? Pharmaceutical companies rely on IMS, grocery stores have Catalina and IRI. Some organization can make the business case to do this on their own, but most can’t.