I have to agree. I used to work for a large IPS vendor and had the
pleasure of taking my product through NSS testing. Now that I don't
work for that employer any more I can honestly say that Bob and his
team at NSS were a pain in the ass to work with, in all the right ways.

Specifically:
1. The tests are well designed to make sure that at least *some*
products could pass. Each year the bar is raised as the products catch
up. For example, at the time it was impossible for any product to
catch 100% of client-side threats and process more than 1 Gbps of
traffic. It was possible to achieve certification without 100% in this
case because otherwise everyone would fail.

2. Yes, vendors could submit one signature content update after the
first round of tests. However, that update must be made available to
all customers within a respectable time period. In addition, if you
read the full report it clearly outlines what the original test
results were and those after the update.

3. Vendors may choose particular signature sets for the testing as
long as they make that config information available to their
customers. Whatever the signature set is, though, the same set is used
for coverage testing and performance testing.

So, they (NSS) may make money from the vendors but they also clearly
recognized that their value is being unbiased. I think that many
vendors would skip NSS testing if they could do so without loss of
customers. It is hard to pass.

No general testing house is going to be perfect for all. Every
customer should evaluate the products based on their own criteria and
not fully trust someone else to do their job for them. However, when
it comes to realistic coverage and performance testing. NSS has struck
the right balance.

One final word, as the poster pointed out, the certification label may
not expire, but I would be wary of a product that does not have a NSS
test report for the current version. That usually indicates they
either didn't have it tested or had it tested and did not want to
release the results.

-J
On Mar 2, 2009, at 10:42 AM, Joel M Snyder wrote:

> I would contend that this is "best of a bad thing."
>
> I have done an enormous amount of testing myself on network security
> products for over 20 years, and Bob Walder's NSS tests are the best
> out there.
>
> The first thing you have to understand is that this kind of testing
> is VERY expensive; it costs a lot of money for the equipment, but it
> costs even more money for the time. Only when a lab like NSS is
> actually getting paid do they have the luxury of doing a very good
> job.
>
> When we test for publications like Network World, we are on a
> dramatically lower budget--we'll test 5 to 10 products for about a
> 10th of what NSS charges to test a single product.
>
> I think that the "certification" thing is a pile of crap (not just
> with NSS, but with every vendor that offers a 'check mark' or 'gold'
> or 'certified' level). However, what comes out of NSS, in addition
> to the useless badges, is an ENORMOUS report based on what they
> actually saw and didn't see. That's the value of their work, and
> that's why I continue to believe that they are the best private test
> lab in our space.
>
> Yes, all of the criticisms you mount (such as the ability of the
> vendor to have a 'do over') are valid, but if you want someone who
> at least has the veneer of independence (despite their being paid by
> the vendor), then the NSS reports are very worthwhile reading.
>
> This may change over time---it's no longer Bob and the South of
> France; it's now a real company in the US with bigger pressures to
> perform. And this is what has caused other previously-reputable
> testers to have lost their reputation.
>
> So, take it with a grain of salt, but anyone who does NOT read the
> NSS reports on products that they have tested is cutting themselves
> off from a huge supply of very high quality data. I won't make that
> statement for most of the other "labs" out there who are doing
> commercial testing.
>
> jms
>
> Disclosure: I've never taken money from NSS, ever. I'm just a fan.
>
>
> Ravi Chunduru wrote:
>> One interesting and provactive slide "Effectiveness" here:
>> http://nsslabs.com/webinars/NSS%20Labs%2010g%20webinar.pdf
>> I agree some what with what was said there, but testing with private
>> exploits alone does not make NSS testing credible. I feel that there
>> are some points which IDP buyers would like to know while selecting
>> the IDP vendor.
>> How many times vendor failed in testing before the product was
>> certified? My understanding is that NSS allows vendors to provide
>> signature pack during testing if it does not meet the pass criteria.
>> Shouldn't this failed number be known to IDP buyers? I also feel
>> that buyers would like to know the Initial coverage number. Without
>> that I don't see the difference between public testing houses and
>> NSS.
>> To make buyers comfortable, I believe testing should be done
>> periodically (Once in a month?) on certified products and take them
>> off the certified list if they don't meet the criteria. I noted that
>> there are some products in the certified list dating back 2004/2005.
>> From the test report, it appears that NSS certifies if 30-40% of
>> client side attacks are detected. Are buyers comfortable with this
>> number?
>> Number of tests made are dismal around 500+. Does that number good
>> enough?
>> Buyers know their internal assets (protocols, applications, operating
>> systems etc..) and would like to see certifications providing
>> detailed
>> information on security effectiveness of common protocols and
>> applications. I don't see these details on NSS reports. I am not
>> sure whether this was the intention of testing by public houses, but
>> one knows clearly on products and their coverage with respect to
>> vulnerabilities and exploits.
>> By the way, are there any testing & certification houses targeting
>> measurement of security coverage with respect to individual protocols
>> servers HTTP, FTP, SSH, SIP, LDAP, SQL Server etc.?
>> Thanks
>> Ravi
>
> --
> Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
> Senior Partner, Opus One Phone: +1 520 324 0494
> jms (at) Opus1 (dot) COM [email concealed] http://www.opus1.com/jms
>
>

0? *?H?÷

 ?0?

10 +0? *?H?÷


 ?t0?-0?? 
 ª:§wÔУ§Ç+i?´O}0
 *?H?÷


0b10 UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA0
080929165059Z
090929165059Z0?10UBennett10
U*Jeremy10UJeremy Bennett1!0 *?H?÷


jeremy (at) deities (dot) org1 [email concealed]0 *?H?÷


jeremyfb (at) mac (dot) com0 [email concealed]?
"0
 *?H?÷



?
0?

?

Å]?!f¦¾ïaW??"`.oß WX#WCõ
1Äk_åàì×|
<µäTRn0?(ÕZh+8ØÓéåkï&Ëêáoõ[g稺=,Á
¶0ªAâ
XéâÎã?½A$Bj±¯d?oß?·âWO©¢ý?A?¼*a?ènӝ££°^ð?ágÝ?ÇO]]?èÖ;?NBv;<ÐÞPL\»À?
K?Pk ?67b-ltÖzÆ0¼î$ó$`)ð}?NhÓ.cZìK?¨««Wû?Ñùð­];hz??CØÎ-LQ
¦K?¼\º}ØEos«;®?g±LZ?ُ

£A0?0/U(0&jeremy (at) deities (dot) org [email concealed]jere
myfb (at) mac (dot) com0 [email concealed]U

ÿ00
 *?H?÷


`ÿp`ª6?ù8?¥z?[w*¨äáY½. Å[rF³ *º³-áþb`½/Vñ/??@¬æø-¢{=Шà¢kqf¶K´^¼b+Ç¢?zÐfÂÞ'@?üM
Ãýå?ê¤bM×/Q¬Æq?7˧º3?hj?Iá~õèøh)*à0??0?¨ 


0
 *?H?÷


0Ñ10 UZA10UWestern Cape10U Cape Town10U
Thawte Consulting1(0&UCertification Services Division1$0"UThawte Personal Freemail CA1+0) *?H?÷


personal-freemail (at) thawte (dot) com0 [email concealed]
030717000000Z
130716235959Z0b10 UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA0?0
 *?H?÷



0?Ä¦<UsUûN¹Ê?ZhÀupßéÿ£ì½Íõ[òv½:aò¿QÎ
ÔåP
0×cZ,?p?ÝÉð+?Zª?qV˯<çñ?6$*Ï+Õó?w=¾+þ»>¿@?dק¦»?eÑÅ*T?H§¶Ñ<
a@dr`·û

£?0?0U

ÿ0

ÿ
0CU<0:08 6 4?2http://crl.tha
wte.com/ThawtePersonalFreemailCA.crl0U
0)U"0 ¤010UPrivateLabel2-1380
 *?H?÷


H?ÑP?ê.Ì
£f¬g¯¬¾Â¡C??L!¸ø6ª-?6/ÀôP ?p<ý­áabÃÙ:~?±?Å?t?%P?bÇÛ'qW%Ý©?9?? Oe_?Ú÷÷?ÖÆN®öê4å[5MwãV!x?Ü!5Þ$±ÓFÿ]_eO1?0?

0v0b10 UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA ª:§wÔУ§Ç+i?´O}0 + ?
o0 *?H?÷

1 *?H?÷


0 *?H?÷

1
090302194107Z0# *?H?÷

1Ed???ݝ¸Wa?uåÔ?Ôõ¬cò0? +

?71x0v0b10 UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA ª:§wÔУ§Ç+i?´O}0?*?H?÷

1x v0b10 UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA ª:§wÔУ§Ç+i?´O}0
 *?H?÷



?
½a>ôx¸ÛúÄÒô'd«8ÈÒâL(ÖLaQà?Äñq""è¡?Jø?U©ªý¸`ªwi-dTHûW"õóI?~Gtâ ?Äã0ÄG???<
ßÙñÖÑÄ<'?衪Ñ???VXÊ2Û/¼ÿ&F0I 3?sa?æ£-©5r?g?®Î??ce?;D?~vÐ?Å:Vb\?ҏc
Øf÷*D,ªËÊΏCäÈÎ?=ZÓ &w8²??ê7KffZý.hqÐ?
Ñ3`EaxjÌà©C=ø]
P´m 44RBhi)½zD9
»z? ½l6Øð"¡
ߺÆ.?aï

[ reply ]