Today, you can’t swing an analytics dashboard online without hitting “big data.” And perhaps that’s no surprise. After all, the benefits gained by winnowing big data into actionable insights are proven –and they’re significant.

It seems there’s just one big question left: Who is going to actually do all that winnowing?

Data analysts and data scientists are a hot commodity right now, given the value they can deliver to companies — especially those overwhelmed by the sheer volume of digital information they’re gathering every minute of every day. However, there aren’t quite enough data pros to fill all the needed positions, and that means opportunities are being lost while companies scramble to fill these crucial roles.

The main challenge is that data scientists have a very unique skill set: an expertise born of education, time, experience and technical prowess. Anyone missing one or two aspects of that skill set may not deliver the complete story companies crave. Even though technology advances are helping make what used to be the sole realm of Ph.D’s more mainstream and intuitive, there are still many open positions seeking data analytics expertise—a trend that doesn’t show signs of reversing anytime soon.

A recent survey sponsored by Teradata —the third “State of Business Intelligence” exploration — took an in-depth look at the importance of big data and the looming talent shortage, and we discovered the following pieces of the puzzle:

  • Companies know big data is important, but unfortunately, they’re not always sure how to get the most out of it. 72 percent of the survey respondents said they want to gain better control over, and understanding of, their data. 66 percent were in favor of putting “softer” data (including social media conversations) to work, too. The numbers split a bit, however, on how to best accomplish these goals, and what tools or staffing you need to make the most of the information available.
  • There is a major shortage of data science talent, and recent tech grads are perpetuating the problem. The survey found that recent information technology grads — the pool from which this position draws — aren’t making data scientist jobs their first priority. Recent graduates have focused on roles in IT or systems analytics (35 percent), program development (32 percent), data management (30 percent) and business analytics (22 percent).
  • The viability and security of data science careers doesn’t match the demand for their talents. Despite being ridiculously in demand, many data scientists leave the field after less than six years on the job. They face concerns about how long their positions will last, what kinds of standards they’re being expected to meet and how much companies are willing to invest in their efforts — and results.

What can be done to fix the situation? Some suggestions:

  • Build a stronger data science community. Whether that means creating more professional resources, offering more opportunities to conduct conversations with other pros or giving more recognition to skilled practitioners, it behooves the big data community to forge a stronger support network.
  • Create a somewhat standardized role, and then explain it to everyone else. It’s difficult to succeed when no one understands thejob you do—and few even try to dig in to find out. By giving employees a greater understanding of the responsibilities of a data scientist, you create more opportunities for collaboration, recognition and appreciation.
  • Start recruiting before graduation. Reaching out to future IT grads with information about data science careers in your company is a great way to draw the talent you need. You could even offer mentoring from data pros within your team, scholarships or tuition reimbursement programs and other resources to ensure they get on the right track—which is, of course, straight to your doors!
  • Invest in employees who show natural skill. Because data science isn’t purely a function of education, you can also foster a crop of smart analysts from within your IT team – or your Marketing Operations team. If you see potential in an employee’s natural bent or they express an interest in big data, why not offer education and other resources to encourage a shift in their career path?

Data scientists are increasingly becoming known as the rock stars of the tech world, but there simply aren’t enough of them to go around. If companies can develop career pipelines that connect more directly to their internal needs, they’ll stand a better chance of fostering talent—and keeping that talent around.