Security Metrics and Risk, How valuable is that dashboard report?

Information security risks are hard to quantify because they involve a lot of “what-if” and “it might happen.” Risks are basically Threats multiplied by Vulnerabilities multiplied by Consequences. Information Security departments use number driven performance dashboards to represent information security risks to a company, or to compliance.

What exactly are these reports saying?
Introducing a little Security Theatre:

<Insert dark and stormy clouds> Our servers are vulnerable, there is going to be a buffer overflow attack. We are going to be pwned!!  The apocalypse is near! Save the women and children. The hackers are gong to get us, they are going to take down our firewall The sys admin’s Fantasy Final Four picks will be stolen!>

Ok, you get the picture. We spend a lot of time postulating on threats, and trying to represent the risk levels to senior management. We try to measure risk by guessing which vulnerabilities will become exploited, and what the consequences could be. We juggle complex technologies and scenarios. Then, we try to squeeze all that complexity into some risk numbers on a pretty dashboard. It gets lost in translation.

It is like giving a tourist to the Middle East a card with ten phases in Arabic on it, and then expecting the tourist to speak and understand Arabic. True risk is lost in translation when we are trying to express complex technological vulnerabilities and threats with number driven metrics. It is like giving senior management a card with ten foreign phrases on it, and expecting them to comprehend a foreign language. Senior executives are being asked to converse and make decisions about something that has really lost its essence. Amit Yoran, CEO of consultancy NetWitness Corp. and former National Cyber Security Division director suggests security resources are often misallocated because organizations are driven to present number-driven metrics based on some combination of threats, vulnerabilities and asset value to management—and that doesn’t work.

“When you try to boil down complex network traffic into a traffic light or some number to present to management–which understands only traffic lights–you’re driving organizations toward bad metrics versus the task at hand,” Yoran said. “We’re struggling to present number-driven metrics to people who struggle to understand all this complexity.” Instead, Yoran suggest that we stop trying estimate the likelihood of threats, and just assume threats are a fact of life for businesses on the Internet. For example, he said there is variance among vulnerability scanners, and scanning the same system with three scanners will render three different sets of results. Vulnerability scanners rely largely on known vulnerabilities and exploits. Vendors have to identify threats and then add the threats to the vulnerability scanner database. So, there is an undetermined amount of lag time between knowledge of a vulnerability and updated signatures on a vulnerability scanner.

Secondly, sophisticated hackers and cybercriminals are not going to use vulnerabilities and exploits that are well known. They are going to use zero-day exploits, rootkits, or vulnerabilities that just have not been made public. Therefore, it is impossible to present an accurate number of threats to an organization.

Yoran suggests that organizations determine the impact of loss of data, rather than concentrate on system or infrastructure security. For too long, he said, security has focused on availability of service rather than focusing on the value of data and keeping it confidential. Our risk analysis should be based on what we are trying to protect instead of “what-if.”

For example, how many of you never locked your car doors until you bought an expensive car with a good stereo system? Instead of locking your door because someone might walk by and take something, you changed your approach to risk by deciding to lock the door and protect the stereo. Yoran is suggesting we examine our data and evaluate risk surrounding that data. Forget about pushing number driven metrics, concentrate on the understanding your company’s data and the risks surrounding that data.

Next post, we will discuss steps to evaluating data and data risks. In the mean time, here are some questions to think about:

What data are you protecting?
How confidential is it?
Is it softball team schedules or personally identifying information (PII?)
How does data flow in and out of your organization?
Where is data stored?