As a data scientist I cannot solve business problems if appropriate data is not available. However, this is the situation faced by security leaders on a daily basis – they have to make strategic decisions, which will impact their cyber security posture, without having access to the insights they need. How can they overcome the twin challenges of a lack of visibility (not enough of the right information) and a lack of confidence (untrusted information)? By developing a robust security metrics programme.
The ability to make informed operational security decisions based on trusted metrics, delivered via automated continuous controls monitoring, could enable security leaders to have confidence that their company and sensitive data is protected. This is evidenced by a July 2019 study, conducted by Forrester Consulting, which surveyed over 250 senior security decision makers in North America and Europe.
Over half (57%) of those surveyed said that having a range of security metrics, such as key performance indicators, key risk indicators, and service-level agreement metrics aligned to any security framework, would be a ‘extremely valuable’ capabilities in a cybersecurity solution.
So automation and a trusted pipeline are needed but what about the metrics themselves? The different types of metrics have different purposes, but if you’re just starting to develop (or refresh) your metrics programme, beginning with metrics that measure your operational security processes is often your best bet.
In my experience, there are four founding principles if you want to develop trusted operational security metrics that are relevant for your organisation:
- Start with process
When starting to develop a metrics program from scratch it’s important to walk before you can run! Often stakeholders (typically CxOs, IT teams, BISOs) find ‘risk metrics’ too abstract to drive action. They may each have different opinions about what ‘risk’ actually is – either because of their role, or their incentives. This results in an uphill battle to create shared understanding of a problem across a diverse group of people, all of whom need to pull in the same direction for security to get stuff done.
A more advisable approach is to start with measuring the performance and efficacy of security process. Making the shift from reporting risk to reporting process goes a long way towards tackling these challenges. You stand a much better chance of creating common ground for a conversation about improving security by delivering insights about how successful a process is right now and what actions can deliver the greatest uplift in effectiveness and/or efficiency.
- Understand the reality of the process
It’s also important to invest time in understanding the reality of the processes being measured. If you don’t understand exactly how the process is applied in an organisation before you build metrics to assess its efficacy, the results will inevitably be misleading, as you’re not measuring what you think you are!
To counter this, examine the whole process and look at the nuances in how things are done by different IT teams who support different platforms in different geographies. Appreciating all this up front and engaging with the people involved will help massively with usability, and therefore acceptance, of the metrics you eventually build.
The other reason for getting closer to the process is so you can develop metrics that are ‘good enough for today’. So rather than over-engineering for an end state on lots of long and tedious conference calls with 20 people that result in zero agreement on what to measure, it’s far more productive to figure out where you are now process-wise and how to get somewhere better in as big a step as is feasible for your organisation – all in a justifiable timescale.
- Create one metric per process
Metrics come into their own when they act first as a tool to help people understand what’s going on, what you need to do to improve, then how to track progress and measure success.
Start by creating one metric per process (think granularly!) – then, if this metric goes out of tolerance, you’ll have a clear idea of how to address this. For example, “number of high severity vulnerabilities” is not easily actionable. If this goes beyond your tolerance what action do you take? It’s affected by multiple processes as well as circumstances beyond your control (patch Tuesday anyone?).
As metrics are developed and you get new views on what your data is telling you, you’ll likely uncover things that are slipping between the cracks of existing processes. If you have a metric that tracks the current status of a given process, but you require several projects to deal with legacy issues that existed before the process was created/updated, there is no harm in splitting that historical data out and tracking those remediation projects separately. This will help you avoid the situation where your metric confuses good current performance with past problems that are now under management via a different – and possibly longer term – process.
This will ultimately allow you to present metrics with more confidence in how they can be improved and on what timescales, as they are directly influenced by specific actions.
- Be clear on the type of process failure
If a metric highlights instances of process failure, figure out if those instances are outliers, or if there is a systemic problem. These two types of issue are likely to require different actions and/or different timescales to address. Of course, we all hope that processes will result in achieving an expected level of security – and that they’ll be followed to the letter. But, where this isn’t the case, we need to dig into why in order to determine our best course of action.
If you follow this approach to develop process-aligned, realistic, granular, actionable metrics you’ll be taking the first step on the journey towards an effective security metrics programme. This will give you visibility and confidence – and that will enable you to have a true impact on the cyber posture of your business.
Dr Leila Powell, Lead Security Data Scientist, Panaseer
