Risk management, policy, compliance and the rest of the ‘boring’ parts of where security collides with business, are all incapable of generating their own atomic metrics of results.
Speaking to security analyst Conrad Constantine, he said that these areas are “just conjecture”, as formalised security monitoring and response can generate detailed metrics for all these processes.
He said: “Do you have a security policy? Are you monitoring for violations of that policy in real-time? Do you have valid business processes that violate that policy because of the nature of the tech and the process? Your policy should serve the business, not the other way around.
“Half the stuff that gets the five-alarms response turns out to be broken business processes, marketing decides to take matters into its own hands and hires some outside company to provide IT services for a short project. This bypasses risk management and suddenly we’re panicking about data heading out to an external party that we’ve never heard of before.”
I asked Constantine if this means that risk management is not being done right. He said that we do ‘risk manage’ in terms of assessing the risk of some change management item or new deployment, and approve or deny it based on the perceived risk, we then never measure the results of taking that risk.
“Here’s my really simple example: some group asks for a new port opened on the firewall. Risk management approves it. Alright.. what now? was that a good idea or not. Well, with a decent monitoring and response workflow in place, you start looking at the number of security alerts that were generated over this opened port, how much analyst time was consumed looking into those?”
He asked how many times does a business make “risk decisions” that end up consuming more time and resources due to unforeseen side effects, and how often are we actually measuring those “side effects” of security investigation work.
“There are continual detection of deviations from compliance, identification of broken and rogue business processes that weaken overall security posture,” he said. “Closed-loop metrics of the security impact of risk decisions to improve business intelligence and efficiency. How many minutes.. hours? of work per year, does the average employee create for the security team?”
I asked him what he meant by “we do risk manage” but never measuring the results of taking that risk. He said that this is one of his “little magnum opus concepts” that he had been working on for years, designing fundamentally different data forensic and investigation response (DFIR) workflow processes, and the software to support them.
He said: “This idea of merging the risk management and DFIR workflows (and accompanying datasets) is something I started on years ago at EMC. For example, department A requests a port change request on a firewall request goes to risk management, they assess the risk and issue a decision. But the results of that decision are never measured further.
“Here’s a simple binding: how many events does the DFIR team see that traverse that newly-opened port? Risk management assesses the probable future of an action, but rarely measures the actual result and learns from them – they follow ‘best practices’ (again as a one-size-fits-all methodology), but nobody seems to be creating data-driven knowledge of how those security decisions /actually play out in their own environment.”
He went o
n to say that security “engineering” is an insult to real engineers as real engineers in any other disciple can simulate the outcomes of their designs. “We don’t do that, security ‘engineering’ is blind faith and ‘best practices’,” he said.
“The only way we can get close to simulation is start out by measuring the results of the decisions we make, with the hope of finding some level of correlation between actions and outcomes.”
I asked Constantine if we are actually measuring those ‘side effects’, and if he could quantify how much time is this creates for the security team per year. “Some of my ballpark figures from EMC came up with a number of approximately three events per year, per employee, for a total analyst-hours investment of about 35 minutes apiece. We do the same kind of metrics for helpdesk/IT support, we need to start figuring this out for security as well,” he said.
“In general, my whole obsession right now is working on ways to lower the bar for effective security responders. As it stands, anyone with less than the ‘10,000 hours’ to expertise is often more of a liability than a benefit.”
He concluded by saying he had led a lot of DFIR teams, and as the resident expert he usually ended up having to do a lot of the junior members jobs for them inevitably, but commoditising the response field to the level where entry level folks can actually be effective was his great white whale these days.
Conrad Constantine was talking to Dan Raywood
