Sunday, December 25, 2011

, ,

Moving to Google App Engine

Sunday, December 25, 2011 Marco de la Vega 22 comments
We are releasing a new VirusTotal version, most of you will not notice much of a change (other than the new layout), nonetheless, it is full of exciting features.  So as to describe them, we are also launching this new blog.

The most noticeable difference is in its backstage, we have moved to Google App Engine. We expect this to bring transparent scalability and high availability (cross your fingers) to VirusTotal. Some months ago we migrated our private API to Google's infrastructure and we could not be happier with the decision, we have forgotten about administration and we can now focus exclusively on coding.

You will also perceive that very often your file uploads will be immediate, this is because we have made use of HTML5 (in those browsers that support it) to compute the hash of the file on the client-side so as to avoid submissions of files that are already present in our store. Additionally, the maximum allowed file size has been increased to 32MB, in coherence with the App Engine's request handler limits.

Regarding the URL scanner, it is probably the VirusTotal feature that has experienced the greatest number of changes:
  • Thanks to App Engine's services the analysis is much faster than before.
  • We have integrated a couple of new engines (VX Vault, SCUMWARE.org, CLEAN MX, etc.) and the total number of scanners now adds up to 19.
  • We added an extended additional information section to the URL reports which includes detailed information returned by the scanning engines (Trend Micro description, Websense ThreatSeeker category, etc.) and by other services that provide information related to the domain/host of the scanned URL (e.g. EXPOSURE).
  • As its predecessor, the new URL scanner also downloads the files (response content) at the scanned URLs, however, this new version will only enqueue for antivirus scanning those files that are not text or similar content (HTML, XML, etc.).
  • With independence of the nature of the response content, the URL scanner will always record the server response headers, this might prove itself useful in tracking the bad guys since very often they will be making use of customised server setups that return certain headers that may be used for fingerprinting.
Neither have we forgotten about the public API and we are releasing its second version, improving the response format so as to be able to add new information to it in the future without having to change the parsers that you might have in place. This new API provides a closer integration with a new version of the private API, so that moving from one to another is far easier than before.

VirusTotal Community has also been subjected to several modifications. You may now vote a file or URL as malicious or harmless without having to comment on it. These votes are used (along with other notions provided by the tools present in VirusTotal) to build a file/URL reputation index that replaces the old safety score. This new index runs from -100 (unanimously malicious) to 100 (unanimously harmless). At the same time, there are no longer standard tags in comments, it must be you (making use of the hashtag - # - symbol) the one that explicitly defines a tag for the comment. Other changes include a new user reputation system that is detailed in the corresponding VirusTotal documentation section.

But not everything have been improvements, unfortunately, in this very first App Engine release the twitter-like public profile comments have been removed and the statistics section has been considerably reduced. The latter is something we pretend to improve over the coming weeks along with other new features. Our roadmap for the near future would, thus, look something like this:
  • Recover the ability to compact VirusTotal reports or transform them to popular formats such as bbcode, HTML, CSV, etc.
  • Include the NSRL file information, which has not been migrated for this very first App Engine release.
  • Improve the statistics section, including not only file scanning indicators but also URL scanning statistics and VirusTotal Community activity.
  • Expand the number of notions in the VirusTotal Community tab so as to create a greater buzz.
  • Include new URL and domain scanners: Malware Domain Blocklist, Palevo Tracker, Malware Patrol, etc.
  • Allow VirusTotal API users to define a URL where their scan results can be posted back as soon as they are available so as to avoid periodic polling for result retrieving.
  • Appoint VirusTotal Community moderators that may ban offensive comments, track down users faking their own or other users' reputation, and ensure the overall quality of the comments in the Community.
  • Build a malware research board that complements VirusTotal Community.
  • Translate the site to as many languages as possible.
And some other features that must remain confidential for the time being which we are completely sure that will delight software developers and site owners.

As usual, we would love to receive your feedback and suggestions, and we hope the new release results in a better VirusTotal experience.

22 comments:

  1. Keep up the great work guys, and I'll keep sending you guys all the malware from my honeypot. :)

    Ken
    Caffeine Security
    http://caffeinesecurity.blogspot.com

    ReplyDelete
  2. I noticed the following behavior:

    1. From Firefox 9.0.1, clicking on the "Choose File" button at https://new.virustotal.com/ does nothing however clicking inside the text box to the left of the button brings up the file selection dialog that I would have expected from clicking on the button.

    2. From IE8, clicking on the "Choose File" button displays the file selection dialog and clicking inside the text box to the left does nothing however double-clicking also brings up the file selection dialog.

    3.I analyzed CCleaner.exe at https://new.virustotal.com/file/7fbc3e3ec53420bb8abb844420ef8349cfa3786243fc76799a72adca32c96c9c/analysis/1325623465/. I would have expected the "thumbs up" count on the right to show 42 rather than zero.

    Thanks for the great work and a fantastic service!

    ReplyDelete
  3. Will existing #malware and #goodware tags be converted to thumbs down and thumbs up in future?

    ReplyDelete
  4. SassDrake, we will be changing the tags to thumbs up and down in the coming days. Any suggestion is always welcome.

    ReplyDelete
  5. Nice work, great site! I noticed that in the conversion the past three weeks of my comments were lost. I'm guessing they are gone forever. Did anyone else report such experience?

    ReplyDelete
  6. Please add a comment preview button at virustotal.org!
    Because the new font system of the messages is messed up and there is no way to edit old postings! x(

    ReplyDelete
  7. One other bug:
    displayed file names are only 24 characters long.

    But also congratulations for the very good new website.

    ReplyDelete
  8. Nice, especially the new file size limit.

    Btw, is it correct that you changed SSL cert?
    old one:
    - Thawte Server CA
    - www.virustotal.com
    SHA1: 9F:7E:98:CC:CA:1A:BA:9A:4A:B0:27:5E:4E:4E:5B:8F:58:83:9E:85
    new one:
    - GeoTrust Global CA
    - RapidSSL CA
    - *.virustotal.com
    SHA1: 14:37:19:71:0D:6D:A6:99:34:C9:79:A4:07:BF:44:94:94:2B:D2:BE

    ReplyDelete
  9. Yes, we wanted a wildcard certificate.

    ReplyDelete
  10. Thank you

    http://www.bilisimplatformu.com/forum/index.php

    ReplyDelete
  11. I like the new design and increased limits. I wish you good coding!

    ReplyDelete
  12. With the new URL scanner is there a way to force the scanning of downloaded HTML pages? Sometimes I would like to check whether the page contains some malicious JavaScript.

    ReplyDelete
  13. Please consider supporting scans within .rar (WinRAR compressed) files.

    ReplyDelete
  14. The Scan URL when supplied with an exe url does not scans with all the antiviruses. This can be really helpful so that user can scan the application without even downloading. Current way to scan an exe is to first download it to computer, upload it again to virustotal and then all the virus scanners scan it.

    It would be really great if Scan URL can understand that the url provided is of an application & the scanning is to perform the application scans

    ReplyDelete
    Replies
    1. Hello, it does make this distinction and it does automatically scan any pointed executables, can you provide me with an example that failed to do so?

      Delete
  15. It's been a while since the last time I submitted malware. I signed in today to upload some fresh samples and imagine my disappointment when I saw my rep points going down from 2000+ to 235. Not sure what went wrong. All the other changes are great and much appreciated.

    ReplyDelete
    Replies
    1. Hello, all reputations where normalized to lower values, having said this, you should be in the same ranking as before, all users experienced the same conversion.

      Delete
  16. thumbs up nice work!

    www.brigade-antivirus.com

    ReplyDelete
  17. Whatever your doing this tool is amazing. It is super clean, looks great, and works flawlessly. We wrote a article rating online files scanners and included your site in our tools a while back. We just updated the article, we need to move your tool up the list now. Great work!

    ReplyDelete