big_data_needle_in_a_haystackJyothi Swaroop is a product marketing executive at a large enterprise technology vendor. He has worked in enterprise software marketing and business development at companies including RainStor (acquired by Teradata) and Oracle.

UnboundID: What are some of the capabilities that enterprises are looking for right now when it comes to supporting consumer big data solutions?

Swaroop: There are four different ones at a high level. The first one is storing data at scale. Companies need to be able to store petabytes of data without spending a ton of money. The second capability is analyzing the data. The third is securing that data and ensuring that it’s not accessed by unauthorized users. Last but not least, is performance. If you run a query against big data, the response needs to come back in time to make a decision that can have an impact on your business.

UnboundID: Are companies prioritizing these capabilities?

Swaroop: This depends on the maturity level of the organization. At an early stage, the CIO might be strictly looking at acquiring efficient storage. If the company is further along in the curve, such as a banking CIO, they already have tons of data in systems like SQL Server and Hadoop. Yet the CIO has huge requirements for security and will need to refine the big data platform further to make it perform faster and more securely.

UnboundID: What are the greatest challenges here in moving from concept to reality, in terms of generating business return?

Swaroop: The whole promise of big data is finding that needle in the haystack, the golden nugget. It is very time-consuming to find that nugget. You need data scientists and analytic experts and a ton of time and when you’re done, you need to know, was it even worth the effort? Did you spend three times the value of the nugget to get it? To get this right, it’s important to know what you want to achieve with big data. That means understanding which are the top five or 10 metrics that are valuable to the business, otherwise you’re probably just shooting in the dark. Next, there is the concept of high-value versus low-value data. Power users can generate a terabyte of data every few days, from email and social media. I would say that less than 1% of that data is valuable. CIOs need to know what is the high value data such as personally identifiable information (PII) and credit card data and know what to do with that data regarding security and access.

UnboundID: How will the big data landscape evolve in the coming year in terms of new capabilities and innovations that CIOs can put to use within their existing infrastructure?

Swaroop: CIOs are looking for the deluge of big data infrastructure of the last few years to become better, faster, cheaper. So technologies like Hadoop or NoSQL and traditional data warehouses will all live on, but there will be technologies to help process data faster. For example, when Hadoop became mainstream, there was the challenge in that while it had the ability to store large chunks of data, analysts wanted to use the same BI tools they always used, which wasn’t always easy or even possible. So that’s going to improve. Second, there is a huge focus on security. We are just dumping data into Hadoop, or something like it, whether that data is high or low value, new or old. It’s like a roach motel. Then the problem has remained how to manage and query it. If you have credit card data or other PII mingled with low value data like tweets, architecturally that’s not a smart way to store and analyze big data. In the future, there will be better ways to secure data and faster ways to access the data. Security will be more granular. CIOs want to segregate data and secure it to level they wish, and that gives consumers a lot of faith in the infrastructure. I will feel more secure when shopping online if I know my banking data is protected and stored separately.

Learn how you can stay competitive in the face of these five mega trends. Download the eBook: Five Mega Trends Driving Digital Transformation