More and more I hear people discussing coverage in terms of security testing. I am here to give you some bad news. You will rarely get a genuine answer on how much coverage you actually received. It is dependent on approach, methodology, tools and skill set.

51% percent of Wikipedia editors agree, the most common forms of coverage testing are:

• Function coverage - Has each function in the program been executed?
• Statement coverage - Has each line of the source code been executed?
• Condition coverage - Has each evaluation point (such as a true/false decision) been executed?
• Path coverage - Has every possible route through a given part of the code been executed?
• Entry/exit coverage - Has every possible call and return of the function been executed?

Each one of these will either result in too little testing or too much testing. What’s worse: it’s unlikely that any two organizations will ever be able to measure this in a way that even allows you to have a conversation about whether or not effective levels of coverage have been obtained. None of these are going to be effective measures of how well your software has been tested. All are, however, guaranteed to increase the amount of time you spend testing.

The problem with security testing is that the devil really is in the details. And there are enough of them that the traditional QA coverage models mentioned above aren’t really effective.

For security testing, lets add:

• Input Coverage - Has every input (e.g. form field, packet fields) been tested?
• Vuln. Class Coverage - Has every form of vulnerability been tested?
• Threat Based Coverage - Has every threat evaluated?

By combining these two, maybe you have an answer that means something. It is obviously still incomplete. There are still application, network and host state that all impact each of the above tests. Also, there are attacks that specifically relate to state and not inputs.

Now we have arrived at one of the places where the security testing world differs from the QA testing world. For the average application, you can make certain assumptions about the environment it will run in to guide the likelihood of certain states. But in the security testing world, an attacker is actively trying to induce any form of state that will cause an advantage.

So, let me ask the readers of this blog some questions:

• What is an acceptable level of coverage in a security test?
• And if you happen to own security somewhere, what would it take for you to actually find a coverage % credible? I am going to guess the M word will rear its ugly head.
• Do you ever have anything that you can even come close to measuring? There are so many states inside of real world applications, even pen test specific forms of coverage aren’t going to come close to being complete.
• If yes, can you effectively convey that to anyone in a way that will actually give them some level of assurance (the a-word of computer security)?

Caution: I am not actually saying, “Don’t try.” All I’m really saying is, “Measuring this stuff is hard, and the amount of time to do it in a credible way is probably best spent on actually testing more.”