Wednesday, 29 August 2012

AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination


[originally written in 2007 (deprecated & offline blog), I have recovered it because remains a topical issue]

I have read today this piece of news:
"An experiment conducted at the end of March by independent security-industry benchmark website VirusTotal.com attempted to simulate a malicious attack using a long-known source of malicious code on computers. Competing with 32 rivals, only Finjan's Vital Security Web Appliance detected and blocked the malicious code in VirusTotal's tests. The computers running other products were all comprised [sic] - resulting in potential data loss and theft."
This paragraph may lead to confusion, whether that was the result intended or not, and that is why we feel compelled to declare the following at VirusTotal:

  • VirusTotal has not conducted any experiment or test related to AV comparative analyses.
  • VirusTotal has no notice whatsoever of the malicious code they refer to in this piece of news.
  • VirusTotal has never tested nor tried Finjan's security solutions.

Generally speaking, even though it may seem obvious, we must state that all anti-malware products have detection problems due to the tremendous proliferation and diversification of malware nowadays. Likewise, any product may detect a new sample on its own, either because of its heuristics or because they are the first ones to generate a specific signature. This is why it seems totally inadequate and opportunistic to claim the superiority of a product based on the result of a sole malware sample.

We are rather tired of repeating that VirusTotal was not designed as a tool to perform AV comparative analyses, but as a tool that checks suspicious samples with several AV programs and helps AV labs by forwarding them the malware they failed to detect. Those who use VirusTotal to perform AV comparative analyses should know that they are making many implicit errors in the methodology, the most obvious being:

  • VirusTotal AV engines are commandline versions, so depending on the product, they will not behave quite like the desktop versions: for instance, in such cases when desktop solutions use techniques based on behavioral analysis and count on personal firewalls that may decrease entry points and mitigate propagation, etc.
  • In VirusTotal desktop-oriented solutions coexist with perimeter-oriented solutions; heuristics in this latter group may be more aggressive and paranoid, since impact of false positives is less visible in the perimeter. It is simply not fair to compare both groups.

In general, it is not an easy task to perform a responsible and reliable AV comparative analysis; it requires having a malware collection that is both representative (nowadays it should be larger than the In-The-Wild collection) and authentic (ZOO collections are riddled with false viruses and corrupt executables). Besides, given the implementation of new AV technologies, in the case of desktop AV products, it would be necessary to execute those samples one by one in real environments with each of the resident products to see their detection capabilities and their prevention. As of today, there is no AV comparative analysis in the world that meets these basic requirements.

4 comments:

  1. If you want, you can recover all of your blog by subscribing to your blog: http://blog.hispasec.com/virustotal/rss20.xml on Google Reader. It keeps a copy of everything.

    ReplyDelete
  2. 蟻力神:http://www.hakanpo.com/p/pro132.html
    三便宝:http://www.hakanpo.com/p/pro149.html
    威哥王:http://www.hakanpo.com/p/pro96.html
    巨人倍増:http://www.hakanpo.com/p/pro138.html
    花痴:http://www.hakanpo.com/p/pro312.html
    D10催情剤:http://www.hakanpo.com/p/pro282.html
    蟻力神:http://www.hakanpo.com/p/pro132.html
    三便宝:http://www.hakanpo.com/p/pro149.html
    D10媚薬:http://www.hakanpo.com/p/pro282.html
    SPANISCHE FLIEGE D5:http://www.hakanpo.com/p/pro308.html
    紅蜘蛛:http://www.hakanpo.com/p/pro311.html
    曲美:http://www.hakanpo.com/p/pro25.html
    妖姫:http://www.hakanpo.com/p/pro481.html
    淫インモラル:http://www.hakanpo.com/p/pro482.html
    インモラル:http://www.hakanpo.com/p/pro482.html
    紅蜘蛛:http://www.hakanpo.com/p/pro311.html
    威哥王:http://威哥王.kanpocom.com
    D10催情剤:http://www.hakanpo.com/p/pro282.html
    花痴:http://www.hakanpo.com/p/pro312.html
    妖姫:http://www.hakanpo.com/p/pro481.html
    淫インモラル:http://www.hakanpo.com/p/pro482.html
    淫インモラル:http://www.hakanpo.com/p/pro482.html
    インモラル:http://www.hakanpo.com/p/pro482.html
    絶對高潮:http://www.hakanpo.com/p/pro133.html

    VigRx:http://www.hakanpo.com/p/pro222.html
    RU486:http://www.hakanpo.com/p/pro113.html
    D10媚薬:http://www.hakanpo.com/p/pro282.html
    三體牛鞭:http://www.hakanpo.com/p/pro331.html
    三体牛鞭:http://www.hakanpo.com/p/pro331.html
    狼1号:http://www.hakanpo.com/p/pro330.html
    狼一号:http://www.hakanpo.com/p/pro330.html
    夜狼神:http://www.hakanpo.com/p/pro574.html
    情愛芳香劑:http://hakanpo.com/p/pro551.html
    アフリカ超人:http://www.hakanpo.com/p/pro573.html
    中華牛鞭:http://www.hakanpo.com/p/pro167.html
    蔵秘雄精:http://www.hakanpo.com/p/pro575.html
    悦可亭:http://www.hakanpo.com/p/pro118.html
    女性用バイアグラ:http://www.hakanpo.com/p/pro579.html
    止血薬カルバゾクロム錠:http://www.hakanpo.com/p/pro580.html
    媚薬:http://www.hakanpo.com/CL05/38.html
    精力剤:http://www.hakanpo.com/CL02
    天天素:http://www.hakanpo.com/p/pro58.html
    リドスプレー:http://www.hakanpo.com/p/pro374.html
    LIDO SPRAY:http://www.hakanpo.com/p/pro374.html
    花之欲:http://www.hakanpo.com/p/pro292.html
    ビグレックス:http://www.hakanpo.com/p/pro386.html
    VVK:http://www.hakanpo.com/p/pro325.html
    男露888:http://www.hakanpo.com/p/pro243.html
    Wenickman:http://www.hakanpo.com/p/pro325.html
    MaxMan:http://www.hakanpo.com/p/pro97.html
    vigrxプラス:http://www.hakanpo.com/p/pro222.html
    モチベーター:http://www.hakanpo.com/p/pro327.html
    勃動力三體牛鞭:http://www.hakanpo.com/p/pro331.html
    中華牛鞭:http://www.hakanpo.com/p/pro167.html
    新一粒神:http://www.hakanpo.com/p/pro476.html
    五便宝:http://www.hakanpo.com/p/pro157.html
    三鞭粒:http://www.hakanpo.com/p/pro450.html
    男宝:http://www.hakanpo.com/p/pro375.html
    魔根:http://www.hakanpo.com/p/pro360.html
    壮天根:http://www.hakanpo.com/p/pro329.html
    黒倍王:http://www.hakanpo.com/p/pro337.html
    九鞭粒:http://www.hakanpo.com/p/pro437.html
    男露888:http://www.hakanpo.com/p/pro243.html
    西班牙蒼蝿水:http://www.hakanpo.com/p/pro284.html
    西班牙蒼蝿粉:http://www.hakanpo.com/p/pro254.html
    性霸2000:http://www.hakanpo.com/p/pro147.html
    Xing霸:http://www.hakanpo.com/p/pro147.html
    情愛芳香劑 :http://www.hakanpo.com/p/pro551.html

    花痴:http://www.hakanpo.com/p/pro312.html
    D10:http://www.hakanpo.com/p/pro282.html
    D8 催情剤:http://www.hakanpo.com/p/pro343.html
    FLY D5:http://www.hakanpo.com/p/pro318.html
    D5原液:http://www.hakanpo.com/p/pro318.html
    蒼蝿水:http://www.hakanpo.com/p/pro318.html
    花之欲:http://www.hakanpo.com/p/pro292.html
    美人豹:http://www.hakanpo.com/p/pro320.html
    福源春:http://www.hakanpo.com/p/pro313.html
    魔鬼天使性欲粉:http://www.hakanpo.com/p/pro309.html
    SEX DROPS:http://www.hakanpo.com/p/pro322.html
    小情人:http://www.hakanpo.com/p/pro322.html
    中絶薬:http://www.hakanpo.com/CL06/42.html
    避妊薬:http://www.hakanpo.com/p/pro117.html
    天天素:http://www.hakanpo.com/p/pro58.html
    超級脂肪燃焼弾:http://www.hakanpo.com/p/pro52.html
    韓国痩身一号:http://www.hakanpo.com/p/pro20.html
    韓国痩身1号:http://www.hakanpo.com/p/pro20.html
    痩身一号:http://www.hakanpo.com/p/pro20.html

    ReplyDelete
  3. シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    シアリス:http://www.cialismall.com
    Cialis:http://www.cialismall.com
    レビトラ:http://www.levitramall.com
    levitra:http://www.levitramall.com
    精力剤:http://www.hikanpou.com
    精力剤:http://www.hakanpo.com
    威哥王:http://威哥王.kanpocom.com
    威哥王:http://xn--kwrt2jp9z.kanpocom.com
    巨人倍増:http://巨人倍増.hikanpou.com
    巨人倍増:http://xn--gmq15an1qquf.hikanpou.com
    紅蜘蛛:http://紅蜘蛛.hakanpo.com
    紅蜘蛛:http://xn--19zl5rhf.hakanpo.com

    ReplyDelete
  4. VigRx:http://www.hikanpou.com/product/VigRx.html
    D10催情剤:http://www.hikanpou.com/product/D10.html
    花痴:http://www.hikanpou.com/product/花痴.html
    威哥王:http://www.hikanpou.com/product/威哥王.html
    巨人倍増:http://www.hikanpou.com/product/巨人倍増.html
    淫インモラル:http://www.hikanpou.com/product/immoral.html
    インモラル:http://www.hikanpou.com/product/immoral.html
    妖姫:http://www.hikanpou.com/product/youhi.html
    RU486:http://www.hikanpou.com/product/RU486.html
    紅蜘蛛:http://www.hikanpou.com/product/紅蜘蛛.html
    D10:http://www.hikanpou.com/product/D10.html
    D10媚薬:http://www.hikanpou.com/product/D10.html
    三鞭粒:http://www.hikanpou.com/product/weige.html
    絶對高潮:http://www.hikanpou.com/product/絶對高潮.html
    FLY D5:http://www.hikanpou.com/product/蒼蝿水.html
    D5原液:http://www.hikanpou.com/product/蒼蝿水.html
    蒼蝿水:http://www.hikanpou.com/product/魔力蒼蝿水5.html
    三便宝:http://www.hikanpou.com/product/三便宝.html
    蟻力神:http://www.hikanpou.com/product/蟻力神1.html
    三體牛鞭:http://www.hikanpou.com/product/三体牛鞭.html
    三体牛鞭:http://www.hikanpou.com/product/三体牛鞭.html
    勃動力三體牛鞭:http://www.hikanpou.com/product/三体牛鞭.html
    狼1号:http://www.hikanpou.com/product/狼1号.html
    狼一号:http://www.hikanpou.com/product/狼1号.html
    五便宝:http://www.hikanpou.com/product/五便宝.html
    男宝:http://www.hikanpou.com/product/男宝5箱.html
    蔵八宝:http://www.hikanpou.com/product/蔵八宝.html
    新一粒神:http://www.hikanpou.com/product/新一粒神.html
    リドスプレー:http://www.hikanpou.com/product/LIDOSPRAY.html
    ビグレックス:http://www.hikanpou.com/product/VigRx.html
    vigrxプラス:http://www.hikanpou.com/product/VigRx.html
    Wenickman:http://www.hikanpou.com/product/Wenickman.html
    MaxMan:http://www.hikanpou.com/product/MaxMan.html
    情愛芳香劑:http://www.hikanpou.com/product/rush724.html
    壮天根:http://www.hikanpou.com/product/壮天根.html
    魔根:http://www.hikanpou.com/product/魔根金中草.html
    夜夜堅:http://www.hikanpou.com/product/夜夜堅.html
    壮陽一号:http://www.hikanpou.com/product/壮陽一号.html
    マカ:http://www.hikanpou.com/product/MACA.html
    黒倍王:http://www.hikanpou.com/product/黒倍王.html
    花之欲:http://www.hikanpou.com/product/花之欲.html
    美人豹:http://www.hikanpou.com/product/美人豹5.html
    妻之友:http://www.hikanpou.com/product/妻之友.html
    福源春:http://www.hikanpou.com/product/福源春.html
    蟻王:http://www.hikanpou.com/product/アリ王蟻王,ANTKING.html
    アリ王:http://www.hikanpou.com/product/アリ王蟻王,ANTKING.html
    SEX DROPS:http://www.hikanpou.com/product/SEXDROPS5.html
    片仔廣:http://www.hikanpou.com/product/片仔廣.html
    へんしこう:http://www.hikanpou.com/product/片仔廣.html
    維尼好:http://www.hikanpou.com/product/維尼好消毒液.html
    縮陰膏:http://www.hikanpou.com/product/縮陰膏.html
    牛黄清心丸:http://www.hikanpou.com/product/牛黄清心丸.html
    同仁堂牛黄清心丸:http://www.hikanpou.com/product/牛黄清心丸2.html
    消渇丸:http://www.hikanpou.com/product/消渇丸.html
    999皮炎平:http://www.hikanpou.com/product//999皮炎平.html
    同仁烏鶏白鳳丸:http://www.hikanpou.com/product/烏鶏白鳳丸.html
    婦科千金片:http://www.hikanpou.com/product/婦科千金片.html
    セフラジン:http://www.hikanpou.com/product/cefradine.html
    終極痩身:http://www.hikanpou.com/product/zhongjishoushen.html
    天天素:http://www.hikanpou.com/product/天天素.html
    スーパーファット:http://www.hikanpou.com/product/スーパーファットNEWSUPERFATBURNIN.html
    SPANISCHE FLIEGE D5:http://www.hikanpou.com/product/SPANISCHE-FLIEGED5.html
    御秀堂:http://www.hikanpou.com/product/御秀堂養顔痩身.html
    韓国痩身一号:http://www.hikanpou.com/product/韓国痩身一号.html
    痩身1号:http://www.hikanpou.com/product/韓国痩身一号.html
    曲美:http://www.hikanpou.com/product/xinqumei.html
    大印象:http://www.hikanpou.com/product/dayinxiang.html
    碧生源:http://www.hikanpou.com/product/bishengyuan.html
    V26:http://www.hikanpou.com/product/V26.html
    避妊薬:http://www.hikanpou.com/pills/避妊中絶.html
    中絶薬:http://www.hikanpou.com/pills/避妊中絶.html
    精力剤:http://www.hikanpou.com/seiryokuzai/精力剤.html
    レディーチャイナ:http://www.hikanpou.com/product/蒼蝿水5.html
    ed治療薬:http://www.hikanpou.com/seiryokuzai/ED治療.html

    ReplyDelete