Power of Big Data is Exciting
Data-driven decision making can give organisations competitive advantage
By Yaki Failtenson, Chief Executive Officer of Varonis www.varonis.com
Varonis published a white paper last year titled, “Mastering the Information Explosion,” examining how quickly data is growing, how the pace of collaboration is accelerating, and how manual methods of managing and protecting data are insufficient. It explained that effective data governance requires harnessing the power of metadata through intelligent automation.
Well, twelve months on and it’s hardly surprising that industry experts are now saying that the same kind of automation is necessary for more than good governance—in order to harness the power of “Big Data,” organizations need to analyze and look for patterns in how and when these massive amounts of data are used, who uses it, in what sequence, and what it contains in order to effectively run a data driven organization.
In “Big Data is Only the Beginning,” Gartner states: “Extreme information management challenges will exacerbate the difficulty of information sharing and will fuel the demand for an overall metadata management capability in enterprises. ”
What does it mean to harness the power of Big Data?
Big data analytics has already turned entire industries on their head. For example, High Frequency Trading (HFT) has completely changed the dynamics of institutional investing. In HFT, trades are executed in microseconds based on huge amounts of information that is processed within seconds of its arrival.
As much as 70% of all trades now are HFT, and is now critical to many firms. HFT has become so effective that it is somewhat controversial and regulators have investigated HFT tactics that might be used to gain an unfair advantage. On the other hand, there is debate over findings from the SEC and CFTC that HFT contributed to volatility during the “flash crash” of May 6, 2010 . Regardless of the veracity of this association, it would be wise to consider that big Data Analysis, if used incorrectly, may lead to a flurry of wrong decisions made very quickly.
A quick internet search for “big data analytics” will return almost 3 million results linking to articles that discuss potential and present success in many fields and verticals—from astrophysics, to healthcare, to finance, to public policy, to retail. There is a lot of excitement and a sense of urgency among executives to make sure their organizations will be ready to compete: In "Executive Advisory: CEO and Senior Executive Survey, 2011; Detail Report," "data-driven decision making" was the technology contribution regarded by CEOs as delivering the most strategic value to the business.”
Big Data analysis and structured data
So far, Big Data analytics has mostly centred on information stores where there is ample metadata to analyze, like websites with extensive logs of activity, and structured data repositories (databases), where transactions are straight-forward to track and analyze. In situations where metadata is available, the challenge truly is about volume and technique—how to process lots of information quickly enough and analyze it effectively to test assumptions, answer questions quickly, detect changes, and understand patterns.
However, Gartner points out, “Business and technologists are realizing that there is even more potential value in evaluating other types of data, some that currently exist in the enterprise and some new types of data. Many organizations have stored data for years and have never attempted to analyze it or look for patterns, simply because the business appetite for doing so didn't exist. ”
Examples of this data include spreadsheets, presentations, images, audio files, video files, blueprints, and designs. This data most often resides in unstructured repositories, like file shares.
Unstructured data repositories often don’t have much existing metadata to analyze. There is usually no record of activity, no strict connection to the creators and owners of the data, and no catalogue or index of what all the data contains. Ironically, this is where the most (and biggest) data actually lives: many studies show that more than 80% of organizational data is stored in unstructured repositories.
Big Metadata – instrumenting unstructured data for Big Data analysis
In ‘Mastering the Information Explosion’, we compare the digital revolution to the transportation revolution; just as more cars and airplanes necessitated traffic lights and air traffic control, more data and collaboration necessitated automated controls for making sure data is correctly accessible and correctly used.
Without automated controls, organizations have found it impossible to identify and track data owners, perform entitlement reviews to manage permissions and maintain a least privilege model, audit data access, spot abuse, and identify stale data. Automated data governance controls are now like the traffic lights in a big city—if you turn them off, everyone needs to drive very slowly or they crash.
The parallel continues: automobile and airplane movements are being tracked and analyzed at scale so we can route our cars with traffic-aware GPS, law enforcement can catch speeders by looking at automated toll-booth records, and air traffic management can make better use of airspace. The vehicles themselves are more sophisticated, providing more information about the status of each component and their overall health so that safety and efficiency increase.
In ‘Pattern-Based Strategy: Getting Value From Big Data’, Gartner writes, “Business leaders place a high priority on the role of technology to deliver meaningful data to the organization, so that they can make better decisions based on facts, rather than assumptions .”
Just as metadata framework technology is now necessary for organizations to manage and protect data residing in unstructured and semi structured repositories, organizations will find that without analyzing metadata, it is impossible to get maximum leverage from their data, to understand its value, identify data sets, correlate them with users, projects, and owners, understand how and when they use their data, where it should be stored, and how they can use it to collaborate more effectively.
The same metadata intelligence and information leverage will be used to enhance business processes along many vectors, optimizing workflows, connecting disparate teams, and discovering new patterns. The good news is that organizations don’t need to change the data storage platforms or their workflows in order to take advantage of this intelligence—they need to instrument the existing the data stores they already use, and use metadata framework technology to normalize, synthesize, and analyze that metadata.
For more information visit www.varonis.com
