I was reading Richard Bejtlich’s blog today on Computer Security Incident Response Teams and he quoted the following from Gartner’s report “How to Build a Computer Security Incident Response Team”:

“A competent and adequately resourced CSIRT is an important part of an organization’s information security program. Many organizations either have nothing in place or follow inconsistent procedures. In many organizations, the goal is to recover from an incident and get back up and running with minimal attention being paid to evidence collection, analysis or postmortem reporting. Over the long term, this approach results in more security events, not fewer, as the organization is unable to discern the root causes of incidents and incorporate these lessons learned into improvements in infrastructure and process management.”

We wholeheartedly agree.  In fact, this is EXACTLY why we released the Verizon Enterprise Risk and Incident Sharing framework (external link) for you to use.  Our hope is that the VERIS framework(1) and our Data Breach Investigations Report(2) is just what you need to mature incident analysis and post-incident reporting.  

The Value of Incident Analysis

“Learn to see in another’s calamity the ills that you should avoid.”

–Thomas Jefferson

We, as an industry, have typically been plagued by a lack of data.   What is worse still is that there’s been quite a bit of nervousness and discussion as to what sort of data is useful in risk management.  Verizon’s RISK Team has long held the belief that some of the most useful information we could study is about our failures. We believe this because examining an incident allows us to have evidence about the state at failure and the degree of failure (impact).  Incident analysis allows us to draw direct evidence about what not to do, and it also allows us to use that evidence to make inferences about that which we should be doing.

As such, this evidence is critical in making decisions.  In fact, our approach to risk is very much influenced by Evidence-Based Medicine (EBM).  For those unfamiliar with the concept, EBM is defined as  “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients (3).”

In fact, Oxford University’s Center for Evidence-Based Medicine suggests that EBM is performed in Five Steps (http://www.cebm.net/?o=1914):

1. Asking Focused Questions: translation of uncertainty to an answerable question
2. Finding the Evidence: systematic retrieval of best evidence available
3. Critical Appraisal: testing evidence for validity, clinical relevance, and applicability
4. Making a Decision: application of results in practice
5. Evaluating Performance: auditing evidence-based decisions

Those readers familiar with the VERIS framework and the DBIR might recognize the RISK Team’s emphasis on applying the first three steps of EBM to these documents.  Indeed, our release of VERIS to the community is an attempt to implicitly engage security practitioners in the first two steps of EBM.  The DBIR represents the act of step 2, it’s release to the public is the act of opening the data set for step 3, so that you can perform steps four and five for your organization.  This process, following steps one through five, is something we call Evidence-Based Risk Management (EBRM).

The Value of Evidence-Based Risk Management

Hopefully the sum of the value of EBRM is immediately evident to you.  It’s a FUD Killer.  EBRM is an antidote to a security industry so out of control that we have “rant blogs” like FudSec that act like an emergency pressure valve for those security professionals that need to let off steam.   The critical thinker will immediately be able to use EBRM to examine claims and assertions.

But in practical application within a security department, especially as that department attempts to align risk exposure and capabilities with the risk tolerance of the data owners (4), EBRM shines in its ability to help make both strategic and tactical decisions. Evidence about impact and maturity can help establish an acceptable risk tolerance range.  Evidence about tactics and techniques, assets and threats helps drive architecture and operational decisions.

Evidence-Based Risk Management – Here To Stay?

Finally, one of the things that makes me optimistic about this concept of EBRM is that I think it’s inevitable.  Unlike other labels we construct and which become fads like GRC or APT, EBRM is simply an expression of scientific method.  It is the desire for proof, data, and indeed, the desire for falsification.  An attack on EBRM is really a defense of ignorance.

Second, drawing from that second value statement for EBRM above, I want to suggest that EBRM has the capability to do what few risk management programs can – be relevant both to security management and to operational security.  Using a VERIS/DBIR EBRM approach already surpasses the usefulness of most risk management standards, as the evidence gathered from incident data is much deeper than they require, much more applicable, and much more “real”.  It is not a poorly fabricated likelihood used to make decisions; it’s the outcome of poorly made decisions already made.

These two qualities suggest, to me at least, a staying power.  A long journey, no doubt, but I hope you’ll join us as we continue on towards evidence and away from fear, uncertainty, and doubt.

(1) https://verisframework.wiki.zoho.com/

(2) http://www.verizonbusiness.com/databreach

(3) Sacket DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. (1996)

Evidence based medicine: what it is and what it isn’t. BMJ 312: 71-2.,

(4) Jack Jones – definition of Risk Management

Tags: DBIR, Evidence-Based Risk Managment, risk, risk management, VERIS

This entry was posted on Tuesday, July 20th, 2010 at 7:59 pm and is filed under 2009 Data Breach Report, Analysis, VERIS. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.