Demonstrating ROI for Penetration Testing (Part Three)

Demonstrating ROI for Penetration Testing (Part Three)
Marcia Wilson 2003-09-11

A brief review

Part one of this series provided a general discussion of ROSI (Return on Security Investment) and likened performing penetration testing to having a health physical. The key idea was to teach security professionals to think like business managers in regards to justifying expenditures for security initiatives and security investments. Part two focused on defining penetration testing as a subset of a security assessment, by introducing information asset valuation and risk management concepts. The main idea was to acknowledge that showing a return on investment for security initiatives such as a Pen Test is a somewhat complicated endeavor. It was suggested that it is possible to justify the expense by aligning with the projected ROI of the project requiring the test. An example of a web-based ERP project was used. Security professionals were encouraged to understand and embrace the language of business rather than rely on the FUD (fear, uncertainty, and doubt) factor. The rub is in understanding the value of Information Assets (IA), which in our example would be a combination of hardware, software, network services, and applications involved in the web-based ERP implementation. The ROI calculations for the project would have been completed prior to the development and implementation of the project including TCO (Total Cost of Ownership) calculations. It was suggested that the Pen Test and results would be a major milestone of the project and would provide a "go" or "no-go" for the project going "live" or being put into production.

Moving forward -- risk analysis

For this part of the series we will focus on defining terms related to the Risk Analysis process, and touch on Information Asset valuation methods. These concepts are critical to understand when justifying the necessity and expense of a Pen Test. Companies want sustainability and survivability. As the Blaster and Sobig worms taught companies in recent weeks, a single worm can shut down a network, congest mail servers, reboot servers, crash desktop systems and wreak havoc in the information systems environment. Loss of productivity for the general employee population, over-time pay for systems and network administrators, data loss, and the impact of loss revenue opportunity will not soon be forgotten. These types of events make justification for security initiatives more readily accepted, but let us not be distracted from the goal and fall back on the FUD factor. What these devastating worms remind us of is that we have valuable Information Assets that are in need of protection in a number of forms. We must have numerous layers of security at the network level, at the operating system level, at the application level, and in our policies, procedures, and behaviors. A Pen Test seeks to penetrate through the layers to expose the weaknesses and vulnerabilities at each layer. For many companies who have not done a formal Business Impact Assessment and Risk Analysis, the events of the past month have automatically highlighted some of the critical Information Assets in need of protection. Those who have suffered could consider the results of the blight as a baseline and work from the information they have at hand.

Wilson's Law

As Metcalf's Law states: As a corporation connects any number (N) of digital devices to its network, (N squared) value accrues to the corporation. And as Murphy's Law states: If anything can go wrong, it will. Let's coin a phrase and call it Wilson's Law: As a corporation connects any number (N) of digital devices to its network, N squared will go wrong and create loss to the corporation. Not particularly clever, but you get the point.

Terms and definitions

Starting from the beginning with our textbook definitions restated: Risk Management involves performing a Risk Analysis, including a cost benefit analysis of the protections deemed necessary to protect Information Assets (IA), and including an iterative process of implementing, reviewing, and maintaining protections. Risk Analysis' goal is to quantify the impact of various threats and vulnerabilities and to put a cost or price tag on the potential loss of business functionality should the threat be realized. Assets are resources, computing infrastructure (hardware, software, networks), processes or products. Anything that threatens the Confidentiality, Integrity, or Availability (CIA) of an asset, whether tangible or intangible is considered a risk. When defining assets, it is important to consider the total cost of ownership for each asset. A threat is an undesirable impact. Vulnerability is the absence or weakness of a protection or safeguard. A safeguard is the control mechanism used to minimize a threat or vulnerability. Along with understanding the correct definitions, security professionals and security management need to embrace the following formulas and definitions:

Exposure Factor (EF)	the percentage of loss that would occur if a particular threat were realized on a particular asset
Single Loss Expectancy (SLE)	the dollar amount that is assigned to one occurrence of a particular event. Asset Value ($) X Exposure Factor = SLE
Annualized Rate of Occurrence (ARO)	the estimated number of times one could expect the threat to occur.
Annualized Loss Expectancy (ALE)	the dollar amount derived from the Single Loss Expectancy times the Annual Rate of Occurrence: SLE X ARO = ALE

Understanding what needs to be protected

Let's bring these formulas home to justifying a Pen Test. In our web-based ERP example, Widget Manufacturing Ltd. has a corporate goal to increase electronic transactions with their suppliers from 30% to 80% within two years. Currently they use an ERP package (SAP) for supplier transactions, however access has only been rolled out to their top three suppliers due to security issues. At present the suppliers must VPN into a secure part of Widget Manufacturing's network, and no web-based "extranet" exists. In order to enable many more (smaller) suppliers to interact with their company, a large Web-based development project linking their ERP package to the Web is in the initial stages. However, there is serious business risk and there are security concerns with this approach as it represents a major change to the way they do business and interact with suppliers. The ROI calculations have already been completed in the original work done to justify the "ERP supplier-extranet initiative". The TCO (total cost of ownership) needs to include the Pen Test and the astute security professional has the task of convincing management that successful completion of the project hinges upon the results of that test. In order for Widget Manufacturing to understand the value of the Pen Test further work is needed.

The asset, or assets, which need protection must be defined and valued. In the case of the above implementation, the project plan has already been put together and therein the differing components for the system have been defined. Some of those components could be:

supplier database(s)
database server(s) (hardware and operating system)
database application (software)
middleware (hardware and software)
web server(s) (hardware and operating system)
web application (software)
network services (router, firewall, load balancing appliance, switches, IDS, Internet connection(s))

Asset inventory and classification precedes valuation, and if the customer does not have a clear picture of the value of each individual asset there are some questions that can be asked that will assist in that process. The military and federal government have a highly defined process of classification for Information Assets, however for our purposes a general discussion ensues. The value of an asset should be based not only on the tangibles (TCO, cost of replacement, cost of downtime), but also on the intangibles that are the hardest to define and include specific skill sets of employees. For the web-based project, if one of the critical components of the system fails, the net effect is that the entire system fails from the perspective of the end-user. Similarly, if one security layer fails the entire system could be put at risk. Depending upon the level of redundancy built into the system, the effect of a component failing varies the net effect of overall failure. Full redundancy is a very good thing, however it heightens the need for security at many levels both vertically and horizontally.

In the knowledge and information age in which we live, the intangibles are the most critical to understand. Here are some questions for the security professional to ask the customer in helping define the value of a penetration test; the goal of which is to uncover possible weaknesses in the design of the overall system.

Which asset is the most critical to the success of the project?
Which asset will generate the most revenue?
Which asset generates the most profitability?
Which asset would be the most expensive to replace?
Which asset would be the most expensive to protect?
Which asset would be the most embarrassing if exposed or cause the greatest liability if revealed?

The value of the penetration test

The database(s) would often be chosen as the most critical of all assets since it contains the information that is sensitive, confidential, and valuable to the organization. A Pen Test would seek to penetrate through the layers of security to reach the database and compromise it in some way. However, as the Pen Tester penetrates layer by layer, the security weaknesses in each layer are documented. If one layer is penetrated, another layer is exposed, and so on and so on. Again, from an end-user perspective, if the entire system doesn't work flawlessly, what good is it? For example, a security breach may occur in that a hacker may have only ("only" being a relative term) knocked the web servers offline via a DOS (denial of service) attack. Did that compromise the most critical asset, the database? No. Did the event make the entire system unusable for the supplier? Yes. Even though the database is the most critical asset, each component of the system becomes critical to the success of the entire system. The value of the Pen Test is to understand where the weaknesses are in the whole system and assist the customer in determining what types of protections, redundancies, and safeguards need to be put in place. Asset identification and valuation is important, but more so is understanding the dependencies between components of the system as a critical next step in the risk analysis process.

Risk analysis requires both quantitative and qualitative information. There are numerous commercial products that automate the risk analysis process, however as mentioned previously, the qualitative data is far more critical and difficult to extract. It often requires face-to-face interviews with key personnel involved in the development, integration, implementation, support and maintenance of the whole system. Once an asset's value has been determined (more on methods later), potential threats can be analyzed (weaknesses and vulnerabilities documented from the Pen Test), and potential loss can be annualized based on how often the threat could occur (based on existing information: CERT, Symantec, @Stake, and experience).

The CxO wants to see some hard dollar figures around the differing scenarios. Taking the security breach example. If the supplier web server(s) are offline for a single day, the organization estimates that the impact of the downtime costs the company $1M per day (SLE) in hardware/software rebuilds, manufacturing delays, loss in productivity, loss in revenue generation, accounting backlogs, etc. Based on existing data, the estimated annualized rate of occurrence (say 50%) multiplied by the single incident cost is equal to the Annualized Loss Expectancy (ARO) of $500,000. Are you thinking what I'm thinking? The organization valued the supplier database as the most critical asset, yet the security incident did not even compromise the database. The incident prevented access to the database, which effectively crippled the supply chain. Are you getting excited about the numbers yet? Valuing assets is not about adding up TCO numbers, replacement costs, etc. Valuing assets is about understanding the impact the loss of a particular asset would have on an organization. This is why Pen Testing is so critical for ensuring the success of bringing business to the web. Information systems are complex and interconnected. Understanding the dependencies between assets and the net effect of a component failure is what Pen Testing is all about.

The need for meaningful measures

There is a tremendous need for meaningful measures when it comes to valuing Information Assets and selecting safeguards without "breaking the bank". Layered security can require a tremendous investment. These are the questions companies are struggling to answer in a definitive way:

What exactly are we trying to protect?
Why exactly are we trying to protect it?
How exactly do we best protect it?
What exactly will happen if we don't protect it?
What exactly is all this going to cost?

The Pen Test goes a long way in answering those questions. A well thought out and documented Pen Test can document and identify assets that were "forgotten" or not accounted for, can verify potential threats and vulnerabilities, and can demonstrate the dependencies and therefore the dependent weaknesses in the system.

There are complicated methods for valuing Information Assets, understanding the dependencies, and breaking down complicated information systems. There are also simple methods. One simple approach could begin with an Information Systems Inventory spreadsheet. Add a few extra columns with one that specifies the purpose of each asset, one column for dependencies, and one column to rate the value of the asset to the organization (low, medium, high). How do you rate the value? Ask what the impact to the organization would be if the asset were lost, corrupted, or stolen? Yes, it is much more complicated than that. There are books written on this topic, but this is a good start. It's a baseline from which to begin. Selecting safeguards based on the value of the asset to the business is not easy either. The dependencies of the systems determine the selection of safeguards.

There is a team-based method for making complex decisions that can be used in selecting safeguards and protections for complex information systems. It is called the Analytical Hierarchical Process (AHP). Developed by Thomas Saaty, AHP provides a way to deal with complex decision making by capturing both subjective and objective evaluation measures, providing a way to check for consistency of the evaluation measures and alternatives. It is useful for making complex decisions involving multiple criteria. An organization can decompose the components of a whole system into individual Information Assets (IA) using this method that comprises the goal, criteria and alternatives and allows for assigning a relative weight to each component (the weight being significant when determining the impact the loss of an asset can have on the business). Comparative choices can be made based on relative scoring.

Summary

For the security practitioner, it is enough to understand that complex methods exist and should be reviewed. The good news is that large organizations are collecting threat and vulnerability information and publishing those results so that the frequency and types of attacks can be understood. It is nice to be able to have some numbers to reference. It is also good news that the security community works together in a very cohesive way when it comes to Internet threats. Measurement is not going to be the problem in the near future. The biggest problem we face is the complexity and interdependency of interconnected global systems.

In the fourth and final article of this series, we will walk through a Pen Test process and discuss types of security safeguards that could be used for Widget Manufacturing, and make final assertions about how ROSI can be proved.

Author Credit

View parts one and two of this series by Marcia J. Wilson on SecurityFocus.