AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

[originally written in 2007 (deprecated & offline blog), I have recovered it because remains a topical issue]

I have read today this piece of news:

"An experiment conducted at the end of March by independent security-industry benchmark website VirusTotal.com attempted to simulate a malicious attack using a long-known source of malicious code on computers. Competing with 32 rivals, only Finjan's Vital Security Web Appliance detected and blocked the malicious code in VirusTotal's tests. The computers running other products were all comprised [sic] - resulting in potential data loss and theft."

This paragraph may lead to confusion, whether that was the result intended or not, and that is why we feel compelled to declare the following at VirusTotal:

• VirusTotal has not conducted any experiment or test related to AV comparative analyses.

• VirusTotal has no notice whatsoever of the malicious code they refer to in this piece of news.

• VirusTotal has never tested nor tried Finjan's security solutions.

Generally speaking, even though it may seem obvious, we must state that all anti-malware products have detection problems due to the tremendous proliferation and diversification of malware nowadays. Likewise, any product may detect a new sample on its own, either because of its heuristics or because they are the first ones to generate a specific signature. This is why it seems totally inadequate and opportunistic to claim the superiority of a product based on the result of a sole malware sample.

We are rather tired of repeating that VirusTotal was not designed as a tool to perform AV comparative analyses, but as a tool that checks suspicious samples with several AV programs and helps AV labs by forwarding them the malware they failed to detect. Those who use VirusTotal to perform AV comparative analyses should know that they are making many implicit errors in the methodology, the most obvious being:

• VirusTotal AV engines are commandline versions, so depending on the product, they will not behave quite like the desktop versions: for instance, in such cases when desktop solutions use techniques based on behavioral analysis and count on personal firewalls that may decrease entry points and mitigate propagation, etc.

• In VirusTotal desktop-oriented solutions coexist with perimeter-oriented solutions; heuristics in this latter group may be more aggressive and paranoid, since impact of false positives is less visible in the perimeter. It is simply not fair to compare both groups.

In general, it is not an easy task to perform a responsible and reliable AV comparative analysis; it requires having a malware collection that is both representative (nowadays it should be larger than the In-The-Wild collection) and authentic (ZOO collections are riddled with false viruses and corrupt executables). Besides, given the implementation of new AV technologies, in the case of desktop AV products, it would be necessary to execute those samples one by one in real environments with each of the resident products to see their detection capabilities and their prevention. As of today, there is no AV comparative analysis in the world that meets these basic requirements.