It’s not easy to find someone with a Ph.D. in Statistics who can sway a boardroom like an esteemed CEO, yet that’s what we expect from data scientists who work with big data. In fact, this role is so complex that Gartner says it’s best suited to a multiperson team.
Along with building models to find patterns within data, data scientists must share their insights with business leaders. They must extract insight from a mine of data, then show stakeholders how to act on it.
These skills don’t come cheap: Experienced data scientists can cost a cool salary of $250,000 or more. GetApp surveyed nearly 500 business leaders to learn how big data is used across several industries. (You can learn more in our methodology section.)
How big data is used on today’s IT teams
Our research found that when businesses employ at least one person to manage their data full-time, they are much more confident that they have the right data and insights to make business decisions.
A data scientist’s role overlaps with IT, so it’s not surprising that our respondents working in IT/tech are most likely to have data scientists and use data science methods in their work. But here’s the catch: Having a data science team doesn’t translate to instant success.
GetApp’s survey showed that IT/tech teams:
- Are most likely to have data scientists
- Find data visualization more helpful than other industries
- Are most likely to say they have the right data but aren’t drawing the right insights from it
Our results suggest that IT/tech teams try to substitute qualitative research with big data. This strategy is bound to fail: Although data reveals key business details, it’s not enough on its own.
As part of our survey, we asked respondents from five sectors if they have employees in their organizations with roles devoted to working with and analyzing business data. Respondents in IT/tech were most likely to say that they have two or more people working in such roles:
But having a data team yields its own concerns. IT/tech is also the sector that’s most likely to stumble when it tries to find insights within the data it has:
GetApp’s research supports a key problem with how big data is used: Quantity doesn’t correlate with quality.
Unfortunately, our survey suggests that IT/tech employees in charge of big data place too much emphasis on quantitative numbers and too little on what motivates people to make certain choices.
The good news is that it’s not an either/or decision: IT/tech teams can change how big data is used to find more effective results.
Don’t expect data to answer the ‘why?’
GetApp’s survey found a positive correlation between using business intelligence (BI) software and higher rates of confidence in decision-making. BI tools ranging from data mining to predictive analytics help users spot patterns, visualize trends, and learn which customers are more or less likely to do certain things.
BI software excels at visualizing the “what” and the “how”— but it rarely explains the “why?”
To gain more value from your BI tools, adjust your expectations of what they can do. They’ll give you the big picture that serves as a first step to start asking the right questions and answer the “why?”
Form a cross-functional working team
When’s the last time you worked with someone outside your immediate IT/tech team? If you’re unsure, that’s a sign of too many silos within your business.
Most of today’s business opportunities exist between teams rather than within them. You’re more likely to find the answer to your “why?” if you work with colleagues in marketing, sales, product, and more.
To change how big data is used in your business, form a cross-functional working team. The group should have a cultural broker from each department, and each cultural broker should have access to your BI app stack.
Conduct qualitative research
Once your team is in place, discuss the most effective ways to engage your customers. Perhaps your sales team spends all day on the phone discussing customer pain points, or your colleague on the product team has a high number of requests for a certain feature.
It’s true that business leaders expect skilled data science teams to spot patterns and explain their business impact. But if data scientists don’t collaborate with colleagues on other teams, they won’t have the full picture of what customers want.
To take big data to the next level, don’t try to automate qualitative research. Instead, use a proven research method, like randomized testing with a control group. Then, use your BI software to analyze and visualize the results. You can use that data to inform your business case for next steps.
Automation is most effective when humans work with machines to perform the tasks that they each excel at. Teams that combine the right BI toolkit with cross-functional culture brokers will have the most sway over how big data is used.
In April 2019, GetApp used Amazon Mechanical Turk to survey 488 business leaders. We required respondents to live in North America and be self-employed, employed part-time, or employed full-time to take the survey. Respondents also had to work in a business with 500 or fewer employees. They worked in one of five verticals: Accounting/finance, healthcare/medical professional, IT/tech, marketing, or sales. Of the 488 qualified responses, 168 came from those in the IT/tech profession.