, SecurityFocus 2008-06-09
The disagreement between antivirus software makers and the firms that test their products took a dramatic turn last week when one company, Trend Micro, told reporters it would boycott a popular certification in the future.
The Cupertino, Calif. company will no longer seek the VB100 certification, Raimund Genes, chief technology officer for Trend Micro, told SecurityFocus on Friday. The certification, which is administered by antivirus-industry watcher Virus Bulletin, tests whether antivirus software can detect a small set of viruses encountered by experts on the Internet, known as the WildList, without flagging non-viruses as malicious.
The problem, Genes said, is that the certification ignores the fact that, at present, the most significant threats are not viruses, but Trojan horse programs and bot software. In addition, antivirus software updates itself over the Internet, but the testing does not allow Internet connectivity for safety reasons, Genes said. Finally, companies have to deal with a massive influx of new threats every year -- in 2007, the number of malware variants topped 500,000 -- but the test only checks for fewer than 1,000 threats.
"I'm okay being tested against a million pieces of malware, because it gives me an idea of where the product stands," Raimund Genes, chief technology officer for Trend Micro, told SecurityFocus. "But I am not okay being tested against 700 outdated pieces of malware."
In response to Trend Micro's statements, Virus Bulletin underscored that the VB100 certification is not intended to measure a product's performance against the largest number of computer threats, but to set a baseline for antivirus products to regularly exceed.
"We have a simple test requirement of detecting the full WildList without false positives," John Hawes, technical consultant for Virus Bulletin, said in an e-mail interview with SecurityFocus. "By monitoring a product's performance over time our results should give an idea of the ongoing competency and reliability of the vendor."
If a product fails to detect any of the viruses from the WildList, which included 678 viruses in April, or categorizes a non-virus as a threat, the software does not receive certification. While vendors occasionally miss spotting a tricky signature or mistakenly flag a harmless file as a virus, regularly passing the test "is the mark of a well-maintained product," Hawes said.
The pass-fail nature of the test has angered software companies in the past. Most notably, antivirus firm Panda has not tested for certification since 2002. Trend Micro has failed to pass its last three VB100 certification tests, starting with a single case of mistaking a non-malicious file as a virus -- known as a false positive -- in August 2007 and culminating in missing three WildList viruses and two false positives in the most recent test in April 2008. Trend Micro and Panda compete with Symantec, the owner of SecurityFocus, in the antivirus software market.
Concerns about the manner in which testing firms evaluate anti-malware software led antivirus companies to make the debate a major topic last year at a conference in Reykjavik, Iceland. In January, security software makers and independent and media-sponsored testing labs agreed to create an industry group the Anti-Malware Testing Standards Organization (AMTSO) to establish best practices and standards in the testing and rating of antivirus software.
The concerns are not new. Writing about the discussions last year, Randy Abrams, director of technical education for antivirus firm Eset, argued that the time has come for antivirus software tests to evolve beyond the WildList.
"Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users," he wrote in June 2007 issue of Virus Bulletin. "The Wild List brought a standard of scientific repeatability and credibility to testers however, if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die."