Big Data is one of those buzzwords you constantly hear about, but many people don’t really know what it is or how it could work for their company. On February 16th, 2012 the New York Times ran an article entitled ‘How Companies Learn Your Secrets’. It told story a Minneapolis man who angrily came into Target one day with a fist full of coupons sent by Target to his high school aged daughter. The coupons were for baby clothes, cribs, etc. and his initial reaction was that Target was trying to encourage his daughter to get pregnant. The manager apologized and sent the man away only to have him return a week later apologize himself when his daughter confessed to being pregnant. How was Target able to determine that his daughter was pregnant before her own father could? They applied predictive analytics on top of the big data purchase history and signals tracked on consumers.

The technology to make these powerful predictions has been quietly developed over the years. Insurance actuaries have used these types of algorithms to predict a variety of risk events. Quants on Wall Street have built predictive models to drive stock trades. Internet examples include Google’s search engine, Adwords, and News Feed, Amazon’s product recommender system, Netflix’s movie service, Facebook’s friend finder and news feed, and Pandora’s radio service. Yahoo also developed this technology for their search engine and display ad service have since released their code to the open source community under the commercially friendly Apache Software License.

With 6 Billion mobile phones in use in 2010, 845 million Facebook users, 500 million twitter users, and more, as well as a projected 40% annual growth in the amount of total data generated across all industries for the foreseeable future. Already, fifteen out of seventeen sectors in the United States have more data stored per company than the US Library of Congress (235 terabytes) according to a June 2011 report by McKinsey Global Institute. Data is being generated today at an exponential rate in comparison with any other time in history. This combined with a significant reduction in the cost of storage and computing power means that companies are deciding to hold on to their data and want use it to drive their businesses. This data deluge is creating the ‘Big Data’ industry which according to the IDC is predicted to grow from $3.2 billion in 2010 to $16.9 billion in 2015.

Infographic by Visual News

Before the use of Big Data is completely main stream, further development is needed in several areas. There are hurdles to overcome for companies when trying to handle the technical challenges of dealing with huge amounts of data. These challenges include data integration, infrastructure, and management. There is development needed to generate insight and value through analytics, data visualizations, and applications of those technologies to address specific industry needs. Finally, there is a need for service providers to help companies harness the power of big data within their organizations. These roles include IT consultants, research companies, management consultants, and marketing companies.

Organizational value will be uncovered through these projects as their beneficiaries realize increased transparency into the signals of their business and the possibility for experimentation on this data. They will be able to segment and take action on subsets of the data, automate those actions for further use, or build decision support systems for human collaboration. Applied machine learning and predictive analytics on this data have applications across industries in areas like customer relationship management, clinical decision support systems, cross-sell and up-sell recommendations, customer retention models, direct marketing, fraud detection, level prediction, and risk management.

GigaOM did a survey of 300 medium to large-sized businesses across North America. 85% of respondents viewed their data as a strategic asset to their business and 77% reported that they had budget allocated for Big Data projects in 2012. A third of those surveyed had put aside 20-30% of their budget for Big Data projects and 57% said this was more budget than a year ago.

The use of big data and predictive analytics is becoming a key way for companies to outperform their peers and its not just for

the Target and Google sized companies anymore. This is a big factor behind the 40% predicted year-on-year industry growth to $16.9 billion in 2015. This is seven times the predicted overall information and communications technology market for the same period. By 2018, it is predicted that this demand growth is going to create a talent supply deficit in the United States which McKinsey Global Institute puts at approximately 140,000-190,000 positions requiring deep expertise in statistics and machine learning and 1.5 million managers and analysts who know how to operate companies by using insights gained from big data. Now is the right time to be getting started and developing in house expertise on Big Data.

Already, more than a half billion dollars of venture investments have been made into Big Data companies and the trend continues. Professional investors in Q4 of 2011 invested another $40M in Cloudera, $11M funding for Kaggle, $15M in second round for DNAnexus, $9.5 series A for Hadapt, $4.2M series B for Digital Reasoning, and $2.5M series A for GridGain. Accel Partners also announced a $100M fund created to invest in applications that help form an ecosystem around existing big data building blocks like Hadoop which was released by Yahoo under the Apache Software License.

Even the White House is getting behind big data. In March of 2012 they announced over $250 million in annual funding for big data projects in the form of grants and department of defense spending. Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy stated, “In the same way that past Federal investments in information-technology R&D led to dramatic advances in supercomputing and the creation of the Internet, the initiative we are launching today promises to transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security.”

Big Data and predictive analytics will continue to take hold and drive the way decisions are made in organizations. Sometimes that will not be transparent as in the case of the Target story and sometimes it will be obvious like with Amazon. Smart companies are realizing the value of their data and making a plan for it now before resources are more scarce and competitive advantage dwindles. The opportunities for small companies to use these technologies to compete with bigger more established companies or create real innovation are real and here to stay.