Focus on Apple
RE: ClamXav for OS X 10.4 Aug 15 2007 09:03PM
David Harley (david a harley gmail com) (2 replies)
RE: ClamXav for OS X 10.4 Aug 16 2007 08:42PM
Todd Woodward (todd_woodward symantec com) (1 replies)
RE: ClamXav for OS X 10.4 Aug 17 2007 09:26AM
David Harley (david a harley gmail com) (1 replies)
RE: ClamXav for OS X 10.4 Aug 17 2007 05:13PM
Dixon, Wayne (wcdixo aurora lib il us) (1 replies)
RE: ClamXav for OS X 10.4 Aug 18 2007 03:23PM
David Harley (david a harley gmail com)
RE: ClamXav for OS X 10.4 Aug 16 2007 02:37PM
William Holmberg (wholmberg amdpi com) (1 replies)
David,

Wonderfully expressed, and to your point:

"I can't understand why so many people think this is a no-brainer.
Literally"

I must say that it is not related only to AntiVirus, but to Security and
general application testing as well.

I was CIO of a testing company for 11 years. One can hardly imagine the
frustrations associated with the people in charge of software efforts
that believe that the programmers can also be the testers, or that a
limited test distributed to potential customers will get the results
they need to put forth a product that is ready for market. Further
frustrating is the media that report on technical items that believe
they understand how to test themselves because they took Stats in
College (Back in "the day")...

When testing virus software, or even just evaluating it for a corporate
purchase, one needs to get as much data about how the tests were
performed, and compare them to each vendors methods. If you don't get
past the marketing hype and into the actual methodology of the test beds
you risk erring on the side of ignorance.

My .02

-Bill

________________________________

From: listbounce (at) securityfocus (dot) com [email concealed] [mailto:listbounce (at) securityfocus (dot) com [email concealed]]
On Behalf Of David Harley
Sent: Wednesday, August 15, 2007 4:03 PM
To: focus-apple (at) securityfocus (dot) com [email concealed]
Subject: RE: ClamXav for OS X 10.4

> For start, I think that it was fairly obvious from everything
> that was written that the 'FightClub' test was not intended
> to be a rigorous piece of science.

Perhaps not. If that -was- the intention, it certainly didn't succeed.
:)

> What was novel was that
> for once the tester tried to comply with one of the key
> requirements, in making his test suite available to everyone,
> so that anyone who wishes to can repeat his testing, analyse
> the samples that were used, and so on.

I don't agree. A key requirement is that the audience knows what the
test suite -is-, either broken down to individual virus names as Virus
Bulletin used to once upon a time (-not- the list of filenames published
in some places), or as related to a known, valid collection. (WildCore,
for instance.) There are ethical and legal problems with making a test
suite available to everyone, but leaving that aside, it's likely that
most people trying to replicate his tests will also replicate his
methods: if they do that, replicating his results won't validate them,
if his methodology was unsound. (Clearly, I think it was!)

Sound testing is difficult to do. It requires knowledge and experience.
As someone who's devoted much of his professional career to learning the
AV craft, it infuriates me that so many people think that it takes no
special skills or knowledge to test as well as or better than
professionals like VB or the guys at Magdeburg.

> It is also relatively
> unusual as the products were used under Linux.

Actually, only some were. Three were appliances and two were Windows
versions.

> Despite florid criticism of the test suite and the
> conclusions of the test, I have yet to see that anyone has
> repeated the tests on the suite used, or on a different
> suite.

I imagine that all the vendors "tested" and most of those that weren't
tried out the test suite. I found some valid files, but some were
0-byte, and some were password-protected zips. The people I've talked to
in that area have found that all the valid samples they tried were
detected by their product. But clearly you're not going to take their
word, or my word for any of this.

> Thus the arguments so far put have been theoretical,
> and unsupported by direct experimental evidence.

You can't have it both ways. If it's unfair to attack Untangled for
being scientific, you can't complain about critics not adhering to
scientific method. :) More to the point, you can't apply true scientific
rigour against a test that's so badly documented.

No-one in the AV testing field is likely to waste time testing properly
on this test suite, if only because it would send the message that a
test suite of 18 files, some of which aren't viruses, is a valid test
suite. Someone I know tried some of them - the presumed viruses, not
EICAR! - out on VirusTotal, and got the interesting result that ClamAV
failed to detect one of the samples Untangled claim it had detected. I'm
not going to draw any conclusions from that. Given the time that's
elapsed since the samples were made available, I'm sure that if any of
the vendors concerned didn't detect any of the samples a week ago, they
certainly do now.

> Several tests that have been conducted away from the
> AV industry's suites and 'labs' have concluded that
> commercial products can perform badly, and that free products
> can perform quite well.

Please. All the tests that are (largely) accepted by the anti-virus
industry are conducted by independent researchers, not the AV industry.
And yes, free products can perform quite well, but they don't compete in
all respects. Not even Disinfectant, a contender for the best freeware
AV of all time. That, in the end, is why John discontinued it, because
people expected more of it than he could deliver on a voluntary project,
as I recall.

> Yet the stock response is often to
> slate the tests, the testing methods, and even the testers,
> rather than to ponder why products do not always perform as
> advertised.

If you can't trust the testing, you can't trust the results. Yes, the
research community is picky. But a lot of people have worked hard for a
long time to establish sound methodologies, only to see one test after
another that assumes that methodology doesn't matter.

> David Harley makes one important issue very clear when he
> considers that "Your test-bed apps have to be similar in
> functionality, and should be configured carefully so that no
> single product has an unfair advantage."
> Whilst I think it valuable in testing for the 'carefully
> configured' setup to be one of the test conditions, in truth
> most users do not spend much time studying the documentation
> and setting an AV product up so carefully.

There's a need for more usability studies, configuration testing and so
on. But this was a detection test.

> One of the most important conditions in any comparative
> review of products is 'as installed', i.e. the manufacturer's
> default configuration, recognising that that is likely in the
> real world to be the way that each product is used. If an AV
> product needs careful tuning and adjusting after
> installation, then few users will ever complete that, and
> some may actual misconfigure in that phase.

That's an issue that deserves closer attention. There is a problem in
that AV tends to prioritise unobtrusive over maximum protection. That
may be what users want, but it's not necessarily what they need. Varies
according to platform, though.

> One of the huge
> industry lessons from Apple that has been largely ignored by
> others is Apple's ability to ship installers that actually do
> an excellent job of configuring quite complex systems first
> off - AirPort Utility being a recent example.

That's a breeze, compared to installing third party security software.

> the complete
> lack of attention that has been devoted to the adverse
> effects of AV products in particular, security and system
> utilities more generally.

That's a fair point. Unfortunately, that's a very different form of
testing from the sort of snapshot testing that detection testing has to
be.

> Currently most Mac users take the attitude that AV products
> have little to offer them, and most can remember being burned
> by commercial products in the past (and a few the free
> protection of Disinfectant).

Let's not forget that Disinfectant -never- offered commercial-grade
protection. And when John found that people insisted on assuming that it
detected macro viruses, which he didn't have the time and resources to
track, he did the honourable thing and sold it to a company who could.
Admittedly they didn't do a very good job with it, but that's a whole
different thread.

As to what AV offers Mac users, I think we've already done that one.

> I do not know what the
> prevailing attitude among Linux users is, but I can imagine
> that enthusiasm for paying for update subscriptions may be
> anathema to many. Perhaps the industry needs to improve still?

The industry is handicapped by the fact that those who work within it
have to eat...

You can't test effectively for everything at once. Don't take this
personally, but that's why so many consumer-level magazine tests are so
poor: not just the variable expertise level, but the fact that the
tester hasn't thought about his targets and how best to achieve them.

Look at the issues Untangle tried (and failed) to tackle in one test:
* ItW testing
* Zoo testing
* Heuristic testing
* EICAR response testing
* configuration testing

Now add some of the additional issues you've raised, whether or not you
put it in these terms:
* usability testing
* default detection testing versus "deep" scanning, high heuristics etc.
* configurability testing (that's not the same as configuration testing)
* misconfiguration testing (!!)
* longitudinal contention/conflict testing
* FP testing
* Disinfection testing

And the issues you didn't mention. (So I won't either: I've already
spent much more time than I can spare on this!)

I can't understand why so many people think this is a no-brainer.
Literally.

--
David Harley
http://www.smallblue-greenworld.co.uk

--
--David Harley

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>

</head>

<body lang=EN-US link=blue vlink=blue>

<div class=Section1>

<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>David,<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Wonderfully expressed, and to your point: <o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>“I can't understand why so many people think this is a
no-brainer. Literally”<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>I must say that it is not related only to AntiVirus, but to Security
and general application testing as well.<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>I was CIO of a testing company for 11 years. One can hardly imagine the
frustrations associated with the people in charge of software efforts that
believe that the programmers can also be the testers, or that a limited test
distributed to potential customers will get the results they need to put forth
a product that is ready for market. Further frustrating is the media that
report on technical items that believe they understand how to test themselves
because they took Stats in College (Back in “the day”)…<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>When testing virus software, or even just evaluating it for a corporate
purchase, one needs to get as much data about how the tests were performed, and
compare them to each vendors methods. If you don’t get past the marketing
hype and into the actual methodology of the test beds you risk erring on the
side of ignorance.<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>My .02<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>-Bill<o:p></o:p></span></font></p>

<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>

<div>

<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>

<hr size=2 width="100%" align=center tabindex=-1>

</span></font></div>

<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>
listbounce (at) securityfocus (dot) com [email concealed] [mailto:listbounce (at) securityfocus (dot) com [email concealed]] <b><span
style='font-weight:bold'>On Behalf Of </span></b>David Harley<br>
<b><span style='font-weight:bold'>Sent:</span></b> Wednesday, August 15, 2007
4:03 PM<br>
<b><span style='font-weight:bold'>To:</span></b> focus-apple (at) securityfocus (dot) com [email concealed]<br>
<b><span style='font-weight:bold'>Subject:</span></b> RE: ClamXav for OS X 10.4</span></font><o:p></o:p></p>

</div>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>

<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>> For start, I think that it was fairly obvious from everything <br>
> that was written that the 'FightClub' test was not intended <br>
> to be a rigorous piece of science. <br>
<br>
Perhaps not. If that -was- the intention, it certainly didn't succeed. :) <br>
<br>
> What was novel was that <br>
> for once the tester tried to comply with one of the key <br>
> requirements, in making his test suite available to everyone, <br>
> so that anyone who wishes to can repeat his testing, analyse <br>
> the samples that were used, and so on. <br>
<br>
I don't agree. A key requirement is that the audience knows what the test suite
-is-, either broken down to individual virus names as Virus Bulletin used to
once upon a time (-not- the list of filenames published in some places), or as
related to a known, valid collection. (WildCore, for instance.) There are
ethical and legal problems with making a test suite available to everyone, but
leaving that aside, it's likely that most people trying to replicate his tests
will also replicate his methods: if they do that, replicating his results won't
validate them, if his methodology was unsound. (Clearly, I think it was!) <br>
<br>
Sound testing is difficult to do. It requires knowledge and experience. As
someone who's devoted much of his professional career to learning the AV craft,
it infuriates me that so many people think that it takes no special skills or
knowledge to test as well as or better than professionals like VB or the guys
at Magdeburg. <br>
<br>
> It is also relatively <br>
> unusual as the products were used under Linux.<br>
<br>
Actually, only some were. Three were appliances and two were Windows versions.<br>
<br>
> Despite florid criticism of the test suite and the <br>
> conclusions of the test, I have yet to see that anyone has <br>
> repeated the tests on the suite used, or on a different <br>
> suite. <br>
<br>
I imagine that all the vendors "tested" and most of those that
weren't tried out the test suite. I found some valid files, but some were
0-byte, and some were password-protected zips. The people I've talked to in
that area have found that all the valid samples they tried were detected by
their product. But clearly you're not going to take their word, or my word for
any of this. <br>
<br>
> Thus the arguments so far put have been theoretical, <br>
> and unsupported by direct experimental evidence.<br>
<br>
You can't have it both ways. If it's unfair to attack Untangled for being
scientific, you can't complain about critics not adhering to scientific method.
:) More to the point, you can't apply true scientific rigour against a test
that's so badly documented. <br>
<br>
No-one in the AV testing field is likely to waste time testing properly on this
test suite, if only because it would send the message that a test suite of 18
files, some of which aren't viruses, is a valid test suite. Someone I know
tried some of them - the presumed viruses, not EICAR! - out on VirusTotal, and
got the interesting result that ClamAV failed to detect one of the samples
Untangled claim it had detected. I'm not going to draw any conclusions from
that. Given the time that's elapsed since the samples were made available, I'm
sure that if any of the vendors concerned didn't detect any of the samples a
week ago, they certainly do now. <br>
<br>
<br>
> Several tests that have been conducted away from the <br>
> AV industry's suites and 'labs' have concluded that <br>
> commercial products can perform badly, and that free products <br>
> can perform quite well. <br>
<br>
Please. All the tests that are (largely) accepted by the anti-virus industry
are conducted by independent researchers, not the AV industry. And yes, free
products can perform quite well, but they don't compete in all respects. Not
even Disinfectant, a contender for the best freeware AV of all time. That, in
the end, is why John discontinued it, because people expected more of it than
he could deliver on a voluntary project, as I recall. <br>
<br>
> Yet the stock response is often to <br>
> slate the tests, the testing methods, and even the testers, <br>
> rather than to ponder why products do not always perform as <br>
> advertised.<br>
<br>
If you can't trust the testing, you can't trust the results. Yes, the research
community is picky. But a lot of people have worked hard for a long time to
establish sound methodologies, only to see one test after another that assumes
that methodology doesn't matter. <br>
<br>
> David Harley makes one important issue very clear when he <br>
> considers that "Your test-bed apps have to be similar in <br>
> functionality, and should be configured carefully so that no <br>
> single product has an unfair advantage." <br>
> Whilst I think it valuable in testing for the 'carefully <br>
> configured' setup to be one of the test conditions, in truth <br>
> most users do not spend much time studying the documentation <br>
> and setting an AV product up so carefully. <br>
<br>
There's a need for more usability studies, configuration testing and so on. But
this was a detection test. <br>
<br>
> One of the most important conditions in any comparative <br>
> review of products is 'as installed', i.e. the manufacturer's <br>
> default configuration, recognising that that is likely in the <br>
> real world to be the way that each product is used. If an AV <br>
> product needs careful tuning and adjusting after <br>
> installation, then few users will ever complete that, and <br>
> some may actual misconfigure in that phase. <br>
<br>
That's an issue that deserves closer attention. There is a problem in that AV
tends to prioritise unobtrusive over maximum protection. That may be what users
want, but it's not necessarily what they need. Varies according to platform,
though. <br>
<br>
> One of the huge <br>
> industry lessons from Apple that has been largely ignored by <br>
> others is Apple's ability to ship installers that actually do <br>
> an excellent job of configuring quite complex systems first <br>
> off - AirPort Utility being a recent example.<br>
<br>
That's a breeze, compared to installing third party security software.<br>
<br>
> the complete <br>
> lack of attention that has been devoted to the adverse <br>
> effects of AV products in particular, security and system <br>
> utilities more generally. <br>
<br>
That's a fair point. Unfortunately, that's a very different form of testing
from the sort of snapshot testing that detection testing has to be. <br>
<br>
> Currently most Mac users take the attitude that AV products <br>
> have little to offer them, and most can remember being burned <br>
> by commercial products in the past (and a few the free <br>
> protection of Disinfectant). <br>
<br>
Let's not forget that Disinfectant -never- offered commercial-grade protection.
And when John found that people insisted on assuming that it detected macro
viruses, which he didn't have the time and resources to track, he did the
honourable thing and sold it to a company who could. Admittedly they didn't do
a very good job with it, but that's a whole different thread. <br>
<br>
As to what AV offers Mac users, I think we've already done that one.<br>
<br>
> I do not know what the <br>
> prevailing attitude among Linux users is, but I can imagine <br>
> that enthusiasm for paying for update subscriptions may be <br>
> anathema to many. Perhaps the industry needs to improve still?<br>
<br>
The industry is handicapped by the fact that those who work within it have to
eat...<br>
<br>
You can't test effectively for everything at once. Don't take this personally,
but that's why so many consumer-level magazine tests are so poor: not just the
variable expertise level, but the fact that the tester hasn't thought about his
targets and how best to achieve them. <br>
<br>
Look at the issues Untangle tried (and failed) to tackle in one test:<br>
* ItW testing<br>
* Zoo testing<br>
* Heuristic testing<br>
* EICAR response testing<br>
* configuration testing<br>
<br>
Now add some of the additional issues you've raised, whether or not you put it
in these terms: <br>
* usability testing<br>
* default detection testing versus "deep" scanning, high heuristics
etc.<br>
* configurability testing (that's not the same as configuration testing)<br>
* misconfiguration testing (!!) <br>
* longitudinal contention/conflict testing<br>
* FP testing<br>
* Disinfection testing<br>
<br>
And the issues you didn't mention. (So I won't either: I've already spent much
more time than I can spare on this!) <br>
<br>
I can't understand why so many people think this is a no-brainer. Literally.<br>
<br>
-- <br>
David Harley<br>
<a href="http://www.smallblue-greenworld.co.uk">http://www.smallblue-greenw
orld.co.uk</a> 
<br>
<br clear=all>
<br>
-- <br>
--David Harley <o:p></o:p></span></font></p>

</div>

</body>

</html>

[ reply ]
RE: ClamXav for OS X 10.4 Aug 16 2007 04:43PM
David Harley (david a harley gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus