A few thoughts for whatever they are worth:

While I agree that deploying products in "real-world" environments is
valuable, I think there is a lot that can be learned by doing tests in
controlled environments as well.

Both are valuable IMO.

Admittedly it's been years since I formally tested any of these
products myself, but in the decade or so of testing I *DID* perform I
have run tests in both real-world scenarios (e.g. The Network
Computing bake-off we did at DePaul University back in 2001) and lab
tests (e.g. subsequent Network Computing tests and the later OSEC
tests from 2001-2004). I - and arguably "we" - have learned from
both.

It has been my experience that if you design the test criteria and
methods properly you can emulate *most* real-world scenarios. For
example, one of the side-effects we found during the DePaul testing
back in 2001 was that not all management protocols were effective in
bandwidth constrained environments; we had placed the sensors down at
the University but the management platforms were back in our lab
connected via a VPN tunnel. Bandwidth was constrained and that
created problems for *some* of the access methods.

Would we have designed a test for that situation before experiencing
it at DePaul? No. Could we emulate and test that scenario now that
we know about it? Certainly.

There are drawbacks to real-world testing, too. For example, we once
ran into a potential hash collision problem in a firewall vendor's
state table management process that we suspect would only creep up in
limited situations - large VOIP deployments being one of them. So if
you did a test in a real-world environment that didn't include VOIP
you probably wouldn't have ever come across the problem...and then
been hosed if you suddenly implemented VOIP and had it going through
this firewall.

How did we find that problem? Synthetic traffic generation. The
chances of us stumbling upon that otherwise were slim to none...

So yeah - there's value in each.

With this latest round of NSS tests, I have yet to see the report (I
don't have an extra $1800 sitting around right now!) but the posted v6
methodology and approach looks pretty solid IMO. If there were
specific flaws in the testing or process I'd like to hear the
complaints specifically, as from my vantage point right now it appears
to be a reasonable approach.

Regardless, like any product evaluation process I think it's valuable
for the reader to:

- consider the criteria and methodology
- consider the author(s) history, reputation, and funding model
- take the results as a single data point
- consider how the results apply to them
- ...come to their own conclusion

Finally - and I'm probably not going to make new friends in saying
this - but if we think any signature-based approach is going to do
anything other than identify really low-hanging fruit we're fooling
ourselves.

If, for example, we integrated custom web applications into the
vulnerability / exploitation test mix we'd probably see a big 0%.
Working for a firm that has actually done the IR work for some of the
breaches we're reading about in the news I can tell you
authoritatively that the exploitation of web application vulns and the
use of custom (often packed) tools will sail past most (all?) of the
commercial IDS/IPS products. Relevant? Depends on who you are, what
you are doing, what data you're protecting...but I digress...

I will say that seeing identification percentages for KNOWN, STOCK
VULNERABILITIES fall below the 50% line is...well...depressing.
Makes you wonder if it's worth investing in the technology, period...

My .02,

-Greg

-----------------------------------------------------------------
Securing Your Online Data Transfer with SSL.
A guide to understanding SSL certificates, how they operate and their application. By making use of an SSL certificate on your web server, you can securely collect sensitive information online, and increase business by giving your customers confidence that their transactions are safe.
http://www.dinclinx.com/Redirect.aspx?36;5001;25;1371;0;1;946;9a80e04e1a
17f194

[ reply ]