As a student of both the fields of Information Technology/Security and Management Science (http://en.wikipedia.org/wiki/Management_science), I often find myself looking at security issues through a “decision-oriented” lens. For the most part, these two disciplines make good bedfellows – especially when one considers that engineers dominate the Information Security field. Please don’t misinterpret this; I have a healthy respect for and advocate our need of engineers (I’ve even helped teach and graduate some of them). However, not all of our problems are engineering problems and I do believe that our ability to truly manage information risk is hindered by a shortage of input from other disciplines (though I’ve seen at least some improvement in recent years).
One area where the engineering and management mindset clash is in decision-making. The engineer asks “What do I need to know to precisely formulate all factors in this decision?” while the management scientist asks “What do I need to know to make a good decision?”. In such matters, I side heavily with the management scientist.
The obvious application of this is in evaluating potential security initiatives or projects (“Should we do X, Y, or Z?”). In most cases, it is impossible to precisely formulate all factors in the decision, so we abandon the “scientific” route and revert to some other method of making it (see below). This is where our predominantly engineering mindset hurts us. Instead, we should realize that organizations have always made decisions using varying amounts of information of varying quality. Our dilemma is not new. Valid and vetted approaches exist for structured decision problems with an abundance of precise data and also for unstructured problems with sparse amounts of “fuzzy” data. They are out there and eagerly waiting for us to apply them to problems in our domain.
Ok, I’m off the soapbox. The main goal of this post is to ask how your company makes “Should we do X, Y, or Z?” decisions. I’ll start the conversation by listing the methods I see used most often. In doing so, I make no judgment on any method’s ability to support good decisions (though it’s clear some have more value than others).
• The “Adamant Auditor” method: You’ve been here. The 22 year old kid shows up 3 months out of the university with his checklist etched in stone. He darn well better be able to check off all those boxes or you’re toast. “But if X does Z and Y does Z, then X=Y… and we’ve done Y” you argue only to receive blank stares. Good luck with that. Unless you can build a credible risk-based argument, you might as well just do X like he says.
• The “Peer Pressure” method: This is the grown-up equivalent to doing what the cool kids do “Peers X and Y are doing Z, so we should too” is the justification here. It might be that X and Y have their act together and are great role models. Then again, they might think that alcohol, blindfolds, and a game of high-speed Chicken make for a great Friday night. Remember what your Mama said – “If so and so jumped off a cliff, would you?”
• The “WIBeHI” method: If you’ve ever used anything that sounds remotely like “Wouldn’t It Be Horrible If X happened, therefore we should do Y” to justify a security initiative, then you’ve used this method. The potential worst-case scenario (and often some extra FUD for good measure) is the main decision criterion in this approach.
• The “Guru Guidance” method: Every organization has its guru and every guru has his opinion. Just ask him. It might be that nobody understands the technical justification behind what they’re recommending, but he knows his stuff, right? Right?
• The “Poll the Panel” method: Often called the “Delphi Method” but I’ve never thought the name very fitting. No journey to a mystical oracle with secret knowledge is required; you simply gather your smart folks and get them to come to a decision. The assumption is that decisions made by many are better than decisions made by one.
• The “Pet Project” method: Perhaps it was the advertisement in that magazine on the plane. Maybe that analyst report. Who knows why your boss wants that project so badly, but its clear she does. And in this job market, who’s going to argue? If you can get it done while also squeezing in something with actual benefit, there’s a chance you can still put a mark in the Win column.
My tone here is obviously facetious but I am quite serious that I believe these methods (or some form of them) account for the majority of security decisions made in most organizations. Is this your experience as well? We’ve put up a quick, one-question poll on the topic here and would love to hear from you (we’ll share the results later). If any of these methods resonate or if you have some to add, please chime in.
As a student of both the fields of Information Technology/Security and Management Science, I often find myself looking at security issues through a “decision-oriented” lens. For the most part, these two disciplines make good bedfellows – especially when one considers that engineers dominate the Information Security field. Please don’t misinterpret this; I have a healthy respect for, and advocate our need of, engineers (I’ve even helped teach and graduate some of them). However, not all of our problems are engineering problems and I do believe that our ability to truly manage information risk is hindered by a shortage of input from other disciplines (though I’ve seen at least some improvement in recent years).
One area where the engineering and management mindset clash is in decision-making. The engineer asks, “What do I need to know to precisely formulate all factors in this decision?” Meanwhile, the management scientist asks “What do I need to know to make a good decision?” In such matters, I side heavily with the management scientist.
The obvious application of this is in evaluating potential security initiatives or projects (“Should we do X, Y, or Z?”). In most cases, it is impossible to precisely formulate all factors in the decision, so we abandon the “scientific” route and revert to some other method of making it (see below). This is where our predominantly engineering mindset hurts us. Instead, we should realize that organizations have always made decisions using varying amounts of information of varying quality. Our dilemma is not new. Valid and vetted approaches exist for structured decision problems with an abundance of precise data and also for unstructured problems with sparse amounts of “fuzzy” data. These approaches are out there and are eagerly waiting for us to apply them to problems in our domain.
Ok, I’m off the soapbox. The main goal of this post is to ask how your company makes “Should we do X, Y, or Z?” decisions. I’ll start the conversation by listing the methods I see used most often. In doing so, I make no judgment on any method’s ability to support good decisions (though it’s clear some have more value than others).
- The “Adamant Auditor” method: You’ve been here. The 22 year old kid shows up 3 months out of the university with his checklist etched in stone. He darn well better be able to check off all those boxes or you’re toast. “But if X does Z and Y does Z, then X=Y… and we’ve done Y” you argue, only to receive blank stares. Good luck with that. Unless you can build a credible risk-based argument, you might as well just do X like he says.
- The “Peer Pressure” method: This is the grown-up equivalent to doing what the cool kids do. “Peers X and Y are doing Z, so we should too” is the justification here. It might be that X and Y have their act together and are great role models. Then again, they might think that alcohol, blindfolds, and a game of high-speed chicken make for a great Friday night. Remember what your Mama said – “If so and so jumped off a cliff, would you?”
- The “WIBeHI” method: If you’ve ever used anything that sounds remotely like “Wouldn’t It Be Horrible If X happened, therefore we should do Y” to justify a security initiative, then you’ve used this method. The potential worst-case scenario (and often some extra FUD for good measure) is the main decision criterion in this approach.
- The “Guru Guidance” method: Every organization has its guru and every guru has his opinion. Just ask him. It might be that nobody understands the technical justification behind what they’re recommending, but he knows his stuff, right? Right?
- The “Poll the Panel” method: Often called the “Delphi Method” but I’ve never thought the name very fitting. No journey to a mystical oracle with secret knowledge is required; you simply gather your smart folks and get them to come to a decision. The assumption is that decisions made by many are better than decisions made by one.
- The “Pet Project” method: Perhaps it was the advertisement in that magazine on the plane. Maybe that analyst report. Who knows why your boss wants that project so badly, but its clear she does. And in this job market, who’s going to argue? If you can get it done while also squeezing in something with actual benefit, there’s a chance you can still put a mark in the Win column.
My tone here is obviously facetious but I am quite serious that I believe these methods (or some form of them) account for the majority of security decisions made in most organizations. Is this your experience as well? We’ve put up a quick, one-question poll on the topic here and would love to hear from you (we’ll share the results later). If any of these methods resonate or if you have some to add, please chime in.
Tags: Decision Making, Information Security, risk management
This entry was posted
on Monday, September 28th, 2009 at 6:27 pm and is filed under Analysis.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
Great post Wade, but…what about the “we have data” method? Is that so impossible?
Posted by: Adam on October 2nd, 2009 at 3:29 pmHi Wade. This is both an interesting post and survey. When I looked at the survey, part of me wanted to select all of them. But then I had to differentiate the possible selections between methods (which they could all be) and what I will call data refinement exercises. Some of the survey selections are from my perspective data refinement activities (poll the panel, wet finger in the air, etc…) that would influence the data going into an actual method (qual, quant, etc..). Some of the other decisions making methods I have seen used on occasion – and maybe loosely accounted for in the survey – are:
Analytical Hierarchy Process – This is very interesting approach for allocating budget for any type of project based off various criteria.
Cost / Benefit Analysis (crude form) – For example, does the cost to mitigate the risk to an acceptable level greater then residual risk itself.
I look forward to the results!
Posted by: Chris Hayes on October 2nd, 2009 at 4:02 pm@Adam: Nope – I don’t think the “we have data” method is impossible. I’m always in favor of that approach, I just don’t see it actually used that often. I think even a little data can be informative enough to bring some sanity to the decision-making process.
Posted by: Wade Baker on October 2nd, 2009 at 7:32 pm@Chris: Hear what you’re saying and mostly agree. The “methods” I listed are often used to collect input to feed qual/quant models but I’ve also seem them used in isolation as the primary decision criterion.
As for Analytic Hierarchy Process, I’m a big fan. In fact, we use that for security project prioritization in one of our client offerings. I’ve also used it to make a personal digital camera decision, which is probably sufficient to initiate me into nerdhood.
Posted by: Wade Baker on October 2nd, 2009 at 7:40 pmYou forgot the executive who’s read a scary story in an airline magazine or the Wall Street Journal and decides “we need to do something about that!” The difference between this approach and WIBeHI is that the security team doesn’t need to do any justification.
Also, there’s “management by crisis exploitation”. When the horrible thing actually happens, a talented CISO will milk it for all it’s worth. The results are good for a few years, but then the commitment fades when there isn’t another crisis. Until after the deterioration is complete, a crisis occurs, and the cycle starts over again.
Posted by: Dean on October 7th, 2009 at 2:43 amWade,
Posted by: Famous Wanderer on October 13th, 2009 at 12:29 pmAnother good thought provoking article. One of the premises in the article is that decisions are made by “the expert” or “the manager”. Some decisions must be made this way for lack of time. But if you have the luxury of time, concensus can be a very effective decision maker but a leader has to have lots of faith to see a complex process play out. So, one approach to good decision making is to involve as many stakeholders as possible. Use inputs from these stakeholders to assess where you are at this particular decision point. This input would probably include CBA and other forms of analytical thinking. Once there is general agreement, then use the brainpower of the stakeholders again (engineers, managers, grey beards and eager new people) to narrow the decision options and then lay out the pros and cons of the essential few paths forward. Most people will be surprised at the excellent judgement of the “herd”. From then on, if you have the luxury of time, you will want lots of inputs on your decision making.