Risk Appetite: Counting Risk Calories is All You Can Do
“If it is impossible to deduce a wave equation strictly logically, then the formal steps carrying on to it, are, as a matter of fact, only witty guesses.” – Max Born
Jim Tiller of British Telecom has published a blog post called “Risk Appetite, Counting Security Calories Won’t Help”. I’d like to discuss Jim’s blog because I think it shows how different our organizations are. I’d also like to counter a few of the assertions he makes because I find these to be misunderstandings that are common in our industry.
“Anyone who knows me or has subjected themselves to my writings knows I have some uneasiness with today’s role of risk. It’s not the process, but more of how there is so much focus on risk as if it were a science – but it’s not. Not even close.”
Let me begin my rebuttal by first arguing that risk management, at it’s basis, is at least ”scientific work”. What I mean by that is elegantly summed up by Eliezer Yudkowsky on the Less Wrong blog. To use Eliezer’s words, I’ll offer that scientific work is “the reporting of the likelihood ratios for any popular hypotheses.”
Our “hypotheses” are simply the statements of likely frequency and impact for various entangled properties of business processes that use computing systems.
And in terms of pointing out differences between what Jim is saying and what Verizon’s Risk Intelligence group believes, not only do we embrace that assertion, but we are actively acting out what Yudkowsky says when he continues that scientists should – “make the actual raw data available, so the likelihoods can be computed for any hypothesis.”
You want “New School Security” or ”Evidence-Based Risk Management”? This, my friends, sums up these notions very succinctly. This is why we have created (and released) VERIS as a framework for creating metrics around security incidents. This is why we release the Data Breach Investigations Report, and why we spend the time and money to work with organizations like the Data Loss Database and the US Secret Service to provide evidence to risk analysts and security professionals.
And know this: we do these things because we believe that Jim is right, a consultant who isn’t engaged in scientific work for their customer is just “guessing” or stating an “opinion”. Even worse, they are doing so without applying rigor to the elimination of bias and without doing all they can to create intersubjectivity between the data owner and the analyst (which is what we really should be doing, not treating “objectivity” as if it is some obtainable state of knowledge).
Let me finish my point here by saying this: You want to know why information risk management isn’t a science yet, Jim? It’s because not enough organizations are following the lead of Verizon, Trustwave, 7safe, the US Secret Service and the Data Loss DataBase (just to name a few) who are actively publishing and sharing information. Frankly, I long for the day when members of our industry (our customers, those we serve) have no tolerance for those who sit and complain about lack of “actuarial quality data” while not doing spit about it.
Speaking of which, the second point I’d like to discuss is that Jim, like many in our industry, assume that there is a magic, happy-place of achievement called “actuarial quality data” and that our inability to accomplish this state of data nirvana prevents us from doing our jobs. In reality, the notion of data quality is made up of subjective elements like “accuracy”, “completeness”, “consistency”, “timeliness”, and so forth. In fact, data quality is the entire reason you have to treat risk management like a science that is heavily dependent on probability theory. The subjectivity in data quality perspectives is best addressed by using the right probabilistic methods.
And make no mistake, Verizon Risk Intelligence isn’t just sitting around waiting for “actuarial quality” to appear at the end of the rainbow. We understand that you can’t achieve if you don’t try, so we continue to make significant investments to increase the accuracy of our data sets based on both those notions of data quality and creating models that do express the uncertainty concerning risk statements.
On to Establishing the Risk Tolerance of an Organization
Really, this isn’t rocket surgery once you understand a couple of significant points.
First, anyone who has studied for the CISSP has had it drilled into them that we serve the data owner. If you want to create intersubjectivity around risk tolerance, the most relevant thing to do is use the tolerance of the data owner. In fact, we might argue that the risk tolerance of any other member of the organization is actually completely irrelevant.
So to use the risk tolerance of the data owner, we have to understand what creates tolerance and intolerance for business risk. In our QRM project prioritization model, for example, our first step is to actually perform interviews with data owners (novel idea, I know). The questions in these interviews have nothing to do with threats, vulnerabilities, or even probable losses from a security incident, but rather help us understand the market conditions within which the company is operating and the business strategy the organization has for profitability (or in the case of non or not-for-profits, the strategy the organization has for maximizing contributions).
Once the basic understanding of market conditions are in place, then we can start to establish the data owner’s tolerance for loss. You can see my post on VERIS impact here for an idea of what sorts of information we look for in this process. Once loss tolerance is established, we now have context within which we can go about creating a state of knowledge for likelihood and impact , complete with reference points for discussion (those reference points being the tolerance for losses, the stated market strategy, and an idea of Total Cost of Ownership for the project(s) under consideration).
Now QRM is only one model we use. We subscribe to the “scientific” ideas of model selection and fit determination. But as customer engagements allow, Risk Intel applies the concepts and processes discussed above in all our engagements.
On “Counting Calories” and the Role/Future Of Risk Management
I’d like to end here with a couple of points. First “risk management” isn’t a fad. Both Jim and Michal Zalewski of Google have recently treated “risk management” as if it were some sort of approach we’re doing for now until we move on to the next great thing. Jim says:
“I must state that this does not mean that risk management is completely pointless — far from it. In lieu of anything better and more accurate, today’s risk processes are what they are.”
I’ll argue that what Jim (and Michal) mean is that the particular risk models they use aren’t accurate enough for their subjective tolerance for uncertainty. Indeed, Michal says that risk = probability of an event * maximum loss. Any risk model that only regards maximum loss is going to be amazingly inaccurate. But where they both make a significant mistake is assuming that risk management is something we’re “trying” or something we do “in lieu of anything better”. Now if you think about it, people have[DH1] been doing risk management since the beginning of history. The current business literature stress on “risk management” as a formalized subject may be a fad, but far after everyone reading post is long dead and gone, people will still be trying to act based on their best perceived likelihood and impact.
If that’s so, we owe it to ourselves to “count calories”. Verizon’s Risk Intelligence believes that this means trying to do the best job we can rather than stating a guess or opinion without rigor.
“If it is impossible to deduce a wave equation strictly logically, then the formal steps carrying on to it, are, as a matter of fact, only witty guesses.” – Max Born
Jim Tiller of British Telecom has published a blog post called “Risk Appetite, Counting Security Calories Won’t Help”. I’d like to discuss Jim’s blog post because I think it shows a difference in perspectives between our organizations. I’d also like to counter a few of the assertions he makes because I find these to be misunderstandings that are common in our industry.
(more…)
