Organisations Today Are Overwhelmed by the Incredible Amount of Data They Collect and Store

Many organisations are sitting on a gold mine of data but do not have effective resources to analyse the data to realise its potential value. Automation and data democratisation is key to solving this.

Today data is everywhere, and more than ever businesses are relying on insights gleaned from data to build better products and services, make smarter business decisions, and compete in their respective industries.

Businesses are collecting an incredible amount of data, but are often struggling to make sense of it all.There are such high rewards for making the right data-driven decisions, but such high consequences for data breaches and for making decisions based off inaccurate or incomplete data.

This has created a tension in the enterprise between business and IT departments which often leads to a tug of war over ownership of corporate data sets and analytics tools.

“Only 4% of companies said they have the right resources to draw meaningful insights from data — and to act on them — Bain & Company

Business departments priority is to leverage corporate data to make better business decisions and uncover opportunities for the organisation. IT departments priority is to keep the data secure, governed, centralised, and ensure compliance with applicable legislation.

These two priorities are often competing with one another — leading to data silos, a bottleneck on IT for access to data with a long backlog, and missed opportunities to make timely data-driven business decisions and uncover new opportunities.

What Is Big Data Anyway and Why Should Companies Care?

Let’s take a step back to take a look at what big data is and what we mean when we say that organisations today are overwhelmed by the incredible amount of data they collect and store.

To help us grasp the scale of data collected and stored today, the rate of new data being created, and the benefits in mining the data let’s take a look at this absolutely amazing infographic courtesy of Forbes.

Why Do Organisations Struggle With Data Analytics?

There is a lot of work that goes into setting up an enterprise data analytics capability, and there is no ‘one size fits all’ solution for enterprises to adopt.

Customers often have to purchase and integrate multiple data analytics products from multiple vendors, which means that large teams are needed to build, integrate and support them all.

A typical enterprise data analytics architecture is broken up into three layers, each of which requiring their own large dedicated teams as illustrated below:

Without investment in automation and data democratisation, the rate at which you can execute on data analytics use cases — and realise the business value — is directly proportionate to the number of data engineers, data scientists, and data analysts you hire.

This scalability issue dramatically increases the cost of data analytics.

This is a big, complex, problem and organisations typically do not have the time or resources to solve this on their own.

What Is Data Democratisation and Why Is It Needed in the Modern Enterprise Today?

“Data democratisation will catapult companies to new heights of performance — if done right” Eric Matisoff, InfoWorld

“Every business is inundated with data from every angle. There is pressure to use insights we glean from the data to improve business performance. As a result of this incredible amount of data to process and new tech that helps non-technical people make sense of the data, there is desire and demand for data democratisation…”

Data democratisation means that everybody has access to data and there are no gatekeepers that create a bottleneck at the gateway to the data. It requires that we accompany the access with an easy way for people to understand the data so that they can use it to expedite decision-making and uncover opportunities for an organization.

The goal is to have anybody use data at any time to make decisions with no barriers to access or understanding.” says Bernard Marr, bestselling author of Big Data in Practice. You can read the original article by Bernard Marr quoted above here. It is an excellent article and Bernard is well respected in the industry so it is well worth your time.

So now that we have an understanding of what data democratisation is, and what the benefits are for an organisation, let’s take a dive into the factors that contribute to achieving data democratisation within an organisation and realising the associated benefits.

Three Main Customer Problems Identified

We tried to get to the root cause of why companies fail to get data democratisation right, and what works for those companies who do manage to get it right.

We looked at how companies are managing their enterprise data analytics functions today and using the architecture pyramid we discussed earlier as a reference point, we uncovered three key elements which need to work together to achieve effective data democratisation within an organisation.

The Key Elements of Data Democratisation

Now that we have uncovered the ‘secret sauce’ of effective data democratisation, the next challenge is how do we implement it. There are only finite human resources available who have the talent to implement each of the key elements, so in order to deliver a scalable solution which can be replicated across many organisations we will need to rely heavily on automation.

Imagine if a company was able to figure out the secret sauce to data democratisation, build a product which automated data democratisation using industry best practise techniques and strategies, and then offered this product as a service for your business.

What would this be worth to you? We went out to the market to work out the scale of the problem facing customers today, how they are currently addressing it, and the value they would place on a solution which solves this problem for them.

Finding Potential Customers To Interview

We set out to validate our understanding of problems facing data analytics customers today.In our previous blog post, we invited you to share your views on the data analytics problems facing companies today, and your thoughts on what is needed in the next generation of data analytics products to address them.

The Response

We interviewed a range of customers to uncover some answers, we have collated our data below.

How Big Is the Problem Facing Customers?

We asked customers to rate on a scale of 1–10, how big the problem is to build, manage, and support each of the three enterprise data analytics tiers for their organisation.

What we found is most customers are relying on humans to solve the problem, who are supported by using tools including Excel and tools provided by the main cloud service providers (GCP, Azure, AWS).

There is no tool on the market today which solves this problem completely, so customers are relying on a combination of people power as well as tools to manage their enterprise data analytics capability.

Customers are hiring specialist teams, either in-house or from a consulting agency, to solve these problems for them.

A common theme we found was that customers said there was a lack of data analytics maturity and skill within their organisation, and that training and change management were large contributing problems for them.

“In the industry I see a lot of problems around change management. Changing infrastructure or platforms is seen as risky. People aren’t taking advantage of the platform and tools to their fullest due to a lack of training and knowledge” – Data Analytics Customer Specialist, Cloud Platform Provider

Another common theme we found customers saying is that is not the initial setup of the infrastructure, tools and data insights which is their biggest problem, but rather it is the ongoing maintenance, change management and ensuring consistency throughout an organisation which is their biggest challenge.

How Well Do Existing Tools Solve the Problem, and How Would Cangler Compare?

We asked customers to rate on a scale of 1–10, how well their existing tools solve their problem and how well Cangler would solve their problem after describing the solution to them.

Across all customer problems, Cangler solved the problem better than the tools on the market today, but we believe we can do even better

What we found was that across all three customer problems, Cangler was viewed as solving the problem better than their current tools. This difference was particularly noticeable in the data acquisition & integration customer problem where Cangler was viewed as being almost 3 times better than the existing tools used today.

We found that the tools on the market today do not adequately solve the problems for customers on their own, so customers are solving this capability gap by hiring third-party data analytics consulting firms to provide the human resources and the expertise to build, operate, and manage their data analytics tools and projects.

According to consultancy.uk, $43 billion was spent on data analytics consulting services in 2017 and more than two thirds (67%) of the executives polled said they expect their organisation to increase analytics consulting spending in future.

Looking at the consulting cost per company, if we assume that a company will hire 5 data analytics consultants to help them that is a cost of $7500 per day ($1500 / day x 5 consultants).

Now if we extrapolate that cost over 261 working days in a year that adds up to a cost of $1,957,500 per year for an organisation in consulting fees alone.

Which companies are working on solutions to solve the problem of data democratisation?

Data democratisation is a relatively new field in data analytics, that said there are already some companies working on developing solutions to address data democratisation in the enterprise such as Databricks, Teradata, Domo, Alteryx, and Cangler.

Each company has their own unique approach to solving the problem. Only time will tell which approach(es) companies will approve of and will adopt as their data democratisation solution. One thing is for sure, the race to solve this problem is on!