Research published today by in-memory analytics database vendor Exasol reveals that dark data remains a fundamental challenge for businesses trying to evolve from business intelligence (BI) to data analytics. Findings show that 82% of organisations do not know where their critical data is located.
The Moving the Enterprise to Data Analytics study, conducted for Exasol by respected research firm Vanson Bourne, looked at how and why organisations in the UK and Germany are transforming from business intelligence to data analytics. Among the findings, 75% organisations confirmed they are moving from BI to data analytics, with the majority regarding themselves as 25%-50% through their transformation process.
Better data (quality, availability and validity) was cited as the main catalyst driving adoption of data analytics within departments, with 62% already realising some of the business value they hoped for. Despite this, businesses are struggling to aggregate and reconcile sources to build the rich datasets that analytics activities require in order to add value beyond BI. Only 11% are seeing their efforts fully pay off. More than half (55%) agree fragmentation of data across multiple databases is slowing that progress.
“Organisations are getting their hands on better data and they see the possibilities, but the data analytics they seek are often out of reach because that data is residing in departmental databases or mounting up in data lakes. They are racing to get to the next stage – becoming more practical and applying that data to business decision making – but most data science teams don’t have the data infrastructure they need to surface that dark data and make data analytics available on demand.” said Mathias Golombek, CTO at Exasol.
The research paints a picture of isolated departmental success, with only 1% of organisations currently regarding themselves as a data-driven enterprise, and just 54% of decisions being data-driven.
The most important reason cited for project failure was security and privacy (29%), impacted by the introduction of new governance practices and uncertainty over the EU’s GDPR coming into force. Not having visibility of critical data – let alone backup and archive data (86%) as well as miscellaneous data (92%) – is a significant and urgent concern. At a time when public awareness and scrutiny concerning privacy is at a high, data reporting and analytics has a crucial role to play.
“GDPR has done the citizen a great service: it’s made organisations look to where their data is and approach it from a human perspective rather than bits and blocks. Businesses that have aggregated data across previously siloed sources so that they can quickly run a subject access request or deletion, for example, have simultaneously created a single view of the customer – the very thing today’s customer centric enterprises are so desperate to achieve. Companies that address the double-edged sword of data analytics to improve customer’s experiences will outperform the old guard,” Golombek added.
The report also found that 40% organisations are already looking at machine learning and AI. At present they lack the volume and quality of data to fully reap its rewards, but as the Moving the Enterprise to Data Analytics report shows, times are moving quickly to adopt data analytics and move towards more proactive business decision making.
