, SecurityFocus 2008-01-25
Security software makers, as well as independent and media-sponsored testing labs, have agreed to create an industry group to establish best practices and standards in the testing and rating of antivirus software, members of the nascent group told SecurityFocus on Thursday.
More than 40 antivirus researchers and security professionals gathered in Bilbao, Spain (corrected), earlier this week to establish the group and discuss guidelines for more rigorously testing antivirus products. The guidelines will focus on documenting the most meaningful ways of testing antivirus products as well as establishing common rules for the data sets which are used for testing, said David Marcus, security research and communication manager at McAfee's AVERT Labs.
"The major concern is that a lot of people will run or try to run antivirus tests, and while we have no issues with people testing our products, we want to question whether you are testing correctly and ... whether you are actually using malware in your test," Marcus said.
The founding of a group focused on testing standards comes 18 months after antivirus companies criticized independent product tester Consumer Reports for grading their products' performance against test data that included 5,500 newly created virus variants. The antivirus companies questioned the reasoning that led to a testing lab writing viruses, while other security researchers argued that it's reasonable to measure the performance of antivirus software against previously unknown threats.
Following the Consumer Reports test and other incidents, anti-virus evaluation firm AV-Test.org and a handful of security firms organized a gathering in Reykjavik, Iceland, last May to discuss guidelines for software tests. During the Antivirus Asia Researchers (AVAR) Conference in Seoul, South Korea, in November, discussions continued, culminating at the Bilbao summit, where antivirus companies and software researchers drew up a charter for the new organization and created committees.
"The plan of action is to meet several times a year and seek agreement on the ways to objectively compare anti-malware security solutions, taking the output of the working committees as working drafts," McAfee researcher Igor Muttik stated on the AVERT blog. "The idea is to have the organization open for the anti-malware companies, the academic institutions, testing bodies, magazine reviewers and everybody, who would wish to participate in improving the standards of testing for security software."
Antivirus firms participating in the new group include F-Secure, Kaspersky, McAfee, Panda Software, and Symantec, which owns SecurityFocus.
The group is not just about responding to the high-profile Consumer Reports' test but the methods of many other reviewers as well, said Andreas Marx, managing director of AV-Test.org, said in an e-mail interview.
"Some testers published results which raised some questions, but nobody wanted to answer them," Marx said. "Some reviews were telling the reader 'your product is bad', but nobody want to answer the questions 'why?' or 'because of what?'"
In addition, many reviews of antivirus software used collections of malicious programs that were not comprehensive or about which the reviewer gave no information. For example, some antivirus tests have used the WildList, a small subset of the current viruses found on the Internet, as their data set, which does not make sense, Marx said.
"The problems we have is that most reviews are focusing on the 'wrong thing,'" he said. "They don't test the products properly."
The inclusion of not just antivirus makers, but software testers and researchers from both industry and academia, should give the group balance, McAfee's Marcus stated. Moreover, while common benchmarks -- most notably in evaluating computer graphics cards -- have led some companies to optimize for the test, rather than the real world, that is less of an issue in antivirus testing, he said.
"You can always bring up the question of whether you are teaching for the test," Marcus said. "That is certainly an understandable concern. But every AV test that we have seen over the past year has had real-world malware."
The group plans to announce its name and participating companies next week.
CORRECTION: The original article misspelled the name of the Spanish city that hosted the recent antivirus testing conference. The name of the city is Bilbao.
If you have tips or insights on this topic, please contact SecurityFocus.