There is, perhaps, a way round these restrictions on publishing benchmarks. Any time you do these benchmarks and the vendor prevents you publishing, amend the article to mention very obviously that the vendor prevented publishing the benchmark. e.g. "upon seeing our benchmark results, vendor foo enacted its non-disclosure clause preventing us from using the data". It doesn't say the figures sucked, but it implies. Vendors who want to compete will be willing to work with the publisher and perhaps tweak the setup to get more favourable results. If the results still stink, then perhaps the product stinks and nobody will buy their products.

In short, if a vendor won't play ball with reviewers, don't buy their product.

One final comment; these kind of restrictions could help along free software some more, as these products don't have any restrictive licensing, allowing fair, unbiased reviews of the products, free from any censorship. This openess should help to ensure a level of trust from customers.

[ reply ]

Link to this comment: http://www.securityfocus.com/comments/columns/55/10053#10053