Yet, Abrams stressed that while there are questions about using the WildList as the sole dataset for the test, the certification is still relevant.

"A single success in passing the Virus Bulletin 100 does not mean anything," Abrams said. "A string of successes typically means that you are paying attention to quality control."

While VB100 certification is a bit like taking a test to which you already had the answer key, companies do fail to get certified occasionally, said Abrams. In the best known recent case, Microsoft failed to gain certification for Windows OneCare in February 2007, but has passed its last two certification tests, according to Virus Bulletin.

"When it happens repeatedly, that gets embarrassing," Abrams said. "When you fail a test, you have customers that call up and ask what is going on."

Abrams' company, Eset, recently boasted in a press release that it had attained its 50th VB100 certification.

Major antivirus-software makers agreed that the testing of their products leaves something to be desired, but none of the companies planned to join Trend Micro in boycotting the VB100 certification. Microsoft stated that the company would continue to test its products for the certification. McAfee, whose VirusScan Enterprise failed its latest test because it missed a single WildList entry, will continue to apply for the certification, a company spokesperson told SecurityFocus.


"The correlation between all tests and real-world performance has dropped in recent years due to the scale of the malware problem," the spokesperson said. "It is unlikely future tests will provide perfect results, but we're working with the anti-malware and testing industry to help improve the situation (as much as) possible."

Symantec, the owner of SecurityFocus, will also continue to submit its products for VB100 certification, the company said in a statement. The security software maker last failed VB100 certification in September 1999.

"Our hope is that current tests will adopt the newer test methodologies on top of their existing test strategies and so give a more comprehensive picture of product protection effectiveness," the spokesperson said.

Virus Bulletin has already taken the industry's criticism to heart and the certification will evolve in the coming months, said Hawes. The WildList, on which the test is based, will begin to include Trojan horse programs encountered in the wild to better reflect the current landscape of threats. However, Virus Bulletin's small size and amount of work required to overhaul the tests have made progress slow, he said.

"We are certainly aware that there are a range of important factors which are not covered by our current testing regime," Hawes said. "Plans to expand and improve the information we offer have been underway for some time."

Until more changes are incorporated into the certification, it's unlikely that Trend Micro will return to testing, the company's Genes said.

"A lot of people have asked them to change, and they haven't," he said. "So we think it is totally meaningless for our customer base to continue testing."

If you have tips or insights on this topic, please contact SecurityFocus.