Focus on Apple
RE: ClamXav for OS X 10.4 Aug 15 2007 09:03PM
David Harley (david a harley gmail com) (2 replies)
> For start, I think that it was fairly obvious from everything
> that was written that the 'FightClub' test was not intended
> to be a rigorous piece of science.

Perhaps not. If that -was- the intention, it certainly didn't succeed. :)

> What was novel was that
> for once the tester tried to comply with one of the key
> requirements, in making his test suite available to everyone,
> so that anyone who wishes to can repeat his testing, analyse
> the samples that were used, and so on.

I don't agree. A key requirement is that the audience knows what the test
suite -is-, either broken down to individual virus names as Virus Bulletin
used to once upon a time (-not- the list of filenames published in some
places), or as related to a known, valid collection. (WildCore, for
instance.) There are ethical and legal problems with making a test suite
available to everyone, but leaving that aside, it's likely that most people
trying to replicate his tests will also replicate his methods: if they do
that, replicating his results won't validate them, if his methodology was
unsound. (Clearly, I think it was!)

Sound testing is difficult to do. It requires knowledge and experience. As
someone who's devoted much of his professional career to learning the AV
craft, it infuriates me that so many people think that it takes no special
skills or knowledge to test as well as or better than professionals like VB
or the guys at Magdeburg.

> It is also relatively
> unusual as the products were used under Linux.

Actually, only some were. Three were appliances and two were Windows
versions.

> Despite florid criticism of the test suite and the
> conclusions of the test, I have yet to see that anyone has
> repeated the tests on the suite used, or on a different
> suite.

I imagine that all the vendors "tested" and most of those that weren't tried
out the test suite. I found some valid files, but some were 0-byte, and some
were password-protected zips. The people I've talked to in that area have
found that all the valid samples they tried were detected by their product.
But clearly you're not going to take their word, or my word for any of this.

> Thus the arguments so far put have been theoretical,
> and unsupported by direct experimental evidence.

You can't have it both ways. If it's unfair to attack Untangled for being
scientific, you can't complain about critics not adhering to scientific
method. :) More to the point, you can't apply true scientific rigour against
a test that's so badly documented.

No-one in the AV testing field is likely to waste time testing properly on
this test suite, if only because it would send the message that a test suite
of 18 files, some of which aren't viruses, is a valid test suite. Someone I
know tried some of them - the presumed viruses, not EICAR! - out on
VirusTotal, and got the interesting result that ClamAV failed to detect one
of the samples Untangled claim it had detected. I'm not going to draw any
conclusions from that. Given the time that's elapsed since the samples were
made available, I'm sure that if any of the vendors concerned didn't detect
any of the samples a week ago, they certainly do now.

> Several tests that have been conducted away from the
> AV industry's suites and 'labs' have concluded that
> commercial products can perform badly, and that free products
> can perform quite well.

Please. All the tests that are (largely) accepted by the anti-virus industry
are conducted by independent researchers, not the AV industry. And yes, free
products can perform quite well, but they don't compete in all respects. Not
even Disinfectant, a contender for the best freeware AV of all time. That,
in the end, is why John discontinued it, because people expected more of it
than he could deliver on a voluntary project, as I recall.

> Yet the stock response is often to
> slate the tests, the testing methods, and even the testers,
> rather than to ponder why products do not always perform as
> advertised.

If you can't trust the testing, you can't trust the results. Yes, the
research community is picky. But a lot of people have worked hard for a long
time to establish sound methodologies, only to see one test after another
that assumes that methodology doesn't matter.

> David Harley makes one important issue very clear when he
> considers that "Your test-bed apps have to be similar in
> functionality, and should be configured carefully so that no
> single product has an unfair advantage."
> Whilst I think it valuable in testing for the 'carefully
> configured' setup to be one of the test conditions, in truth
> most users do not spend much time studying the documentation
> and setting an AV product up so carefully.

There's a need for more usability studies, configuration testing and so on.
But this was a detection test.

> One of the most important conditions in any comparative
> review of products is 'as installed', i.e. the manufacturer's
> default configuration, recognising that that is likely in the
> real world to be the way that each product is used. If an AV
> product needs careful tuning and adjusting after
> installation, then few users will ever complete that, and
> some may actual misconfigure in that phase.

That's an issue that deserves closer attention. There is a problem in that
AV tends to prioritise unobtrusive over maximum protection. That may be what
users want, but it's not necessarily what they need. Varies according to
platform, though.

> One of the huge
> industry lessons from Apple that has been largely ignored by
> others is Apple's ability to ship installers that actually do
> an excellent job of configuring quite complex systems first
> off - AirPort Utility being a recent example.

That's a breeze, compared to installing third party security software.

> the complete
> lack of attention that has been devoted to the adverse
> effects of AV products in particular, security and system
> utilities more generally.

That's a fair point. Unfortunately, that's a very different form of testing
from the sort of snapshot testing that detection testing has to be.

> Currently most Mac users take the attitude that AV products
> have little to offer them, and most can remember being burned
> by commercial products in the past (and a few the free
> protection of Disinfectant).

Let's not forget that Disinfectant -never- offered commercial-grade
protection. And when John found that people insisted on assuming that it
detected macro viruses, which he didn't have the time and resources to
track, he did the honourable thing and sold it to a company who could.
Admittedly they didn't do a very good job with it, but that's a whole
different thread.

As to what AV offers Mac users, I think we've already done that one.

> I do not know what the
> prevailing attitude among Linux users is, but I can imagine
> that enthusiasm for paying for update subscriptions may be
> anathema to many. Perhaps the industry needs to improve still?

The industry is handicapped by the fact that those who work within it have
to eat...

You can't test effectively for everything at once. Don't take this
personally, but that's why so many consumer-level magazine tests are so
poor: not just the variable expertise level, but the fact that the tester
hasn't thought about his targets and how best to achieve them.

Look at the issues Untangle tried (and failed) to tackle in one test:
* ItW testing
* Zoo testing
* Heuristic testing
* EICAR response testing
* configuration testing

Now add some of the additional issues you've raised, whether or not you put
it in these terms:
* usability testing
* default detection testing versus "deep" scanning, high heuristics etc.
* configurability testing (that's not the same as configuration testing)
* misconfiguration testing (!!)
* longitudinal contention/conflict testing
* FP testing
* Disinfection testing

And the issues you didn't mention. (So I won't either: I've already spent
much more time than I can spare on this!)

I can't understand why so many people think this is a no-brainer. Literally.

--
David Harley
http://www.smallblue-greenworld.co.uk

--
--David Harley
> For start, I think that it was fairly obvious from everything <br>> that was written that the 'FightClub' test was not intended <br>> to be a rigorous piece of science. <br><br>Perhaps not. If that -was- the intention, it certainly didn't succeed. :)
<br><br>> What was novel was that <br>> for once the tester tried to comply with one of the key <br>> requirements, in making his test suite available to everyone, <br>> so that anyone who wishes to can repeat his testing, analyse
<br>> the samples that were used, and so on. <br><br>I don't agree. A key requirement is that the audience knows what the test suite -is-, either broken down to individual virus names as Virus Bulletin used to once upon a time (-not- the list of filenames published in some places), or as related to a known, valid collection. (WildCore, for instance.) There are ethical and legal problems with making a test suite available to everyone, but leaving that aside, it's likely that most people trying to replicate his tests will also replicate his methods: if they do that, replicating his results won't validate them, if his methodology was unsound. (Clearly, I think it was!)
<br><br>Sound testing is difficult to do. It requires knowledge and experience. As someone who's devoted much of his professional career to learning the AV craft, it infuriates me that so many people think that it takes no special skills or knowledge to test as well as or better than professionals like VB or the guys at Magdeburg.
<br><br>> It is also relatively <br>> unusual as the products were used under Linux.<br><br>Actually, only some were. Three were appliances and two were Windows versions.<br><br>> Despite florid criticism of the test suite and the
<br>> conclusions of the test, I have yet to see that anyone has <br>> repeated the tests on the suite used, or on a different <br>> suite. <br><br>I imagine that all the vendors "tested" and most of those that weren't tried out the test suite. I found some valid files, but some were 0-byte, and some were password-protected zips. The people I've talked to in that area have found that all the valid samples they tried were detected by their product. But clearly you're not going to take their word, or my word for any of this.
<br><br>> Thus the arguments so far put have been theoretical, <br>> and unsupported by direct experimental evidence.<br><br>You can't have it both ways. If it's unfair to attack Untangled for being scientific, you can't complain about critics not adhering to scientific method. :) More to the point, you can't apply true scientific rigour against a test that's so badly documented.
<br><br>No-one in the AV testing field is likely to waste time testing properly on this test suite, if only because it would send the message that a test suite of 18 files, some of which aren't viruses, is a valid test suite. Someone I know tried some of them - the presumed viruses, not EICAR! - out on VirusTotal, and got the interesting result that ClamAV failed to detect one of the samples Untangled claim it had detected. I'm not going to draw any conclusions from that. Given the time that's elapsed since the samples were made available, I'm sure that if any of the vendors concerned didn't detect any of the samples a week ago, they certainly do now.
<br><br><br>> Several tests that have been conducted away from the <br>> AV industry's suites and 'labs' have concluded that <br>> commercial products can perform badly, and that free products <br>> can perform quite well.
<br><br>Please. All the tests that are (largely) accepted by the anti-virus industry are conducted by independent researchers, not the AV industry. And yes, free products can perform quite well, but they don't compete in all respects. Not even Disinfectant, a contender for the best freeware AV of all time. That, in the end, is why John discontinued it, because people expected more of it than he could deliver on a voluntary project, as I recall.
<br><br>> Yet the stock response is often to <br>> slate the tests, the testing methods, and even the testers, <br>> rather than to ponder why products do not always perform as <br>> advertised.<br><br>If you can't trust the testing, you can't trust the results. Yes, the research community is picky. But a lot of people have worked hard for a long time to establish sound methodologies, only to see one test after another that assumes that methodology doesn't matter.
<br><br>> David Harley makes one important issue very clear when he <br>> considers that "Your test-bed apps have to be similar in <br>> functionality, and should be configured carefully so that no <br>> single product has an unfair advantage."
<br>> Whilst I think it valuable in testing for the 'carefully <br>> configured' setup to be one of the test conditions, in truth <br>> most users do not spend much time studying the documentation <br>> and setting an AV product up so carefully.
<br><br>There's a need for more usability studies, configuration testing and so on. But this was a detection test. <br><br>> One of the most important conditions in any comparative <br>> review of products is 'as installed',
i.e. the manufacturer's <br>> default configuration, recognising that that is likely in the <br>> real world to be the way that each product is used. If an AV <br>> product needs careful tuning and adjusting after
<br>> installation, then few users will ever complete that, and <br>> some may actual misconfigure in that phase. <br><br>That's an issue that deserves closer attention. There is a problem in that AV tends to prioritise unobtrusive over maximum protection. That may be what users want, but it's not necessarily what they need. Varies according to platform, though.
<br><br>> One of the huge <br>> industry lessons from Apple that has been largely ignored by <br>> others is Apple's ability to ship installers that actually do <br>> an excellent job of configuring quite complex systems first
<br>> off - AirPort Utility being a recent example.<br><br>That's a breeze, compared to installing third party security software.<br><br>> the complete <br>> lack of attention that has been devoted to the adverse
<br>> effects of AV products in particular, security and system <br>> utilities more generally. <br><br>That's a fair point. Unfortunately, that's a very different form of testing from the sort of snapshot testing that detection testing has to be.
<br><br>> Currently most Mac users take the attitude that AV products <br>> have little to offer them, and most can remember being burned <br>> by commercial products in the past (and a few the free <br>> protection of Disinfectant).
<br><br>Let's not forget that Disinfectant -never- offered commercial-grade protection. And when John found that people insisted on assuming that it detected macro viruses, which he didn't have the time and resources to track, he did the honourable thing and sold it to a company who could. Admittedly they didn't do a very good job with it, but that's a whole different thread.
<br><br>As to what AV offers Mac users, I think we've already done that one.<br><br>> I do not know what the <br>> prevailing attitude among Linux users is, but I can imagine <br>> that enthusiasm for paying for update subscriptions may be
<br>> anathema to many. Perhaps the industry needs to improve still?<br><br>The industry is handicapped by the fact that those who work within it have to eat...<br><br>You can't test effectively for everything at once. Don't take this personally, but that's why so many consumer-level magazine tests are so poor: not just the variable expertise level, but the fact that the tester hasn't thought about his targets and how best to achieve them.
<br><br>Look at the issues Untangle tried (and failed) to tackle in one test:<br>* ItW testing<br>* Zoo testing<br>* Heuristic testing<br>* EICAR response testing<br>* configuration testing<br><br>Now add some of the additional issues you've raised, whether or not you put it in these terms:
<br>* usability testing<br>* default detection testing versus "deep" scanning, high heuristics etc.<br>* configurability testing (that's not the same as configuration testing)<br>* misconfiguration testing (!!)
<br>* longitudinal contention/conflict testing<br>* FP testing<br>* Disinfection testing<br><br>And the issues you didn't mention. (So I won't either: I've already spent much more time than I can spare on this!)
<br><br>I can't understand why so many people think this is a no-brainer. Literally.<br><br>-- <br>David Harley<br><a href="http://www.smallblue-greenworld.co.uk">http://www.smallblue-greenw
orld.co.uk</a>  <br><br clear="all">
<br>-- <br>--David Harley

[ reply ]
RE: ClamXav for OS X 10.4 Aug 16 2007 08:42PM
Todd Woodward (todd_woodward symantec com) (1 replies)
RE: ClamXav for OS X 10.4 Aug 17 2007 09:26AM
David Harley (david a harley gmail com) (1 replies)
RE: ClamXav for OS X 10.4 Aug 17 2007 05:13PM
Dixon, Wayne (wcdixo aurora lib il us) (1 replies)
RE: ClamXav for OS X 10.4 Aug 18 2007 03:23PM
David Harley (david a harley gmail com)
RE: ClamXav for OS X 10.4 Aug 16 2007 02:37PM
William Holmberg (wholmberg amdpi com) (1 replies)
RE: ClamXav for OS X 10.4 Aug 16 2007 04:43PM
David Harley (david a harley gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus