Showing posts with label API. Show all posts
Showing posts with label API. Show all posts

Thursday, May 13, 2021

, , , , , , , , ,

Compliant, easy and actionable integration of VirusTotal in 3rd-party products - Welcome VT Augment

TL;DR: We are releasing an official, compliant and recommended method for displaying VirusTotal context in 3rd-party products and services, so that end-users can enjoy a single pane of glass experience when working with their tools of choice. Read the docs / See the demo (click on the VirusTotal icon next to each observable).



Security analysts world-wide are demanding a single pane of glass experience from their products. Corporate cybersecurity stacks are increasingly complex: too many tools and services, information scattered across numerous databases, arduous stitching together of disparate sources, etc. Incident response and threat hunting have become a time consuming quest across multiple browser tabs. The experience is poor.

If you develop some kind of security product, you will probably know that a common request coming from users is to integrate VirusTotal threat context and reputation. CrowdStrike can speak to this popular demand, just recently we worked together to build a Falcon-VirusTotal integration for their CrowdStrike store. We will be speaking about this and other integrations with our antivirus partners in future posts.

Notwithstanding, at VirusTotal we have to make sure that our data is not misused to the detriment of the ecosystem, this is why we have a strict policy regarding scanning companies and use of our services. This is also why our premium service terms prohibit displaying raw data in 3rd-party products and interfaces, especially those exposed to end-customers.

This said, over time we have seen many legit use cases for integration, mostly revolving around enrichment (adding a layer of context) of alerts/detections that get generated through some means other than VirusTotal. Indeed, when Incident Responders and SOC analysts review alerts, they want to answer questions such as:
  • Given a hash in an alert, is there any second stage payload that I should be searching for in my environment?
  • What’s the C2 infrastructure tied to a given hash? Has it shown up in my network logs?
  • Given a domain flagged by my IDS, is it a flagrant false positive based on its popularity and malicious observations recorded by VirusTotal?
  • Given an IP address that matched my threat feeds, has it been seen serving malware? If so, which hashes? Have those been seen across my fleet of machines?
  • Once my EDR reveals a compromise, is it a well known threat to the industry? i.e. is it widely detected? Is it rather a targeted attack?
To answer these and other questions many companies have implemented a bring-your-own-api-key model where their end-users plug their VirusTotal API key into their products. Sightings recorded in those products then get automatically enriched via such key. While this is theoretically OK, it has resulted in poor and weak integrations that:
  • Are extremely basic and often just display detection ratios, which is not only on the verge of compliance but is pretty useless given today’s false positive and false negative rates.
  • Fail to display the wealth of contextual information that VirusTotal records: C2s and network traffic, delivery mechanisms, relationships with other artifacts, submission and in-the-wild metadata, crowdsourced detections via {YARA, SIGMA, IDS} rules, etc.
  • Do not evolve as VirusTotal itself improves. Whenever we incorporate new data points or release new features, these rarely show up in those integrations. Moreover, for them to show up the integrator must invest engineering resources to update the logic.
  • Miss the opportunity to create a single product experience where common customers can easily pivot from the 3rd-party product into VirusTotal to conduct deeper investigations.
Not everything is lost. We are introducing VT AUGMENT, an HTML widget that greatly reduces the heavy lifting required to display VirusTotal context in 3rd-party products:
  • It can enrich the most common threat observables: files/hashes, domains, IPs and URLs.
  • You do not need to parse complex API response objects and build fancy templates, VirusTotal directly serves a report with all the context that we have for the observable.
  • The report can be styled to match your interface.
  • VirusTotal seamlessly adds new features and data points to the widget, without requiring engineering work on your side to update.
  • It allows you to display all VirusTotal details, not just a subset of them. Moreover, it is not constrained to an analysis data dump, it also displays our threat graph for the given observable and any related IoCs.

All the details displayed in the report are pivotable, meaning that your users can search for similar files, jump to other files communicating with the same domain, discover other malware signed with the same authenticode certificate, etc. with a single click.



Most importantly, VT AUGMENT technically ensures compliance with our terms. Since we are not exposing parseable API fields, there is no room for backend black magic to drive detections, perform machine learning, copy signatures or other non-compliant use cases that would go against the VirusTotal ecosystem.

You can see VT AUGMENT in action in the following demo environment that simulates a SIEM alerts dashboard, click on the VirusTotal icon next to each observable to display the VT AUGMENT widget:
https://www.virustotal.com/ui/widget/demo/dedicated?full=1

Please note that VT AUGMENT still requires you to implement a bring-your-own-api-key model where your end-users plug their API key into your product. This said, we are also open to consider integrations driven by a single integrator key, always with prior consent from VirusTotal.



You can dive into the specifics surrounding the VT AUGMENT integration in our API reference:
https://developers.virustotal.com/v3.0/reference#widget-overview

A standard VirusTotal API key will be enough to test the flow, but remember that the final setup must make use of each of your users’ API keys, unless you have explicit permission from VirusTotal. Additionally, note that we have also published a handy javascript client library to further ease the task of displaying the widget report in your own interface:
https://github.com/VirusTotal/vt-augment

If you are interested in integrating VT AUGMENT and require more API quota to develop and test the setup, need some help with the technical implementation or want to discuss partnership opportunities involving this new widget please do not hesitate to contact us.

We will be creating a specific section in our main VirusTotal website to document and showcase these integrations. Similarly, we will be showing some gratitude to the first adopters by featuring their integrations in an upcoming blog post series on this topic, stay tuned!

Wednesday, November 06, 2019

, , , , , , , , , ,

Pipelining VT Intelligence searches and sandbox report lookups via APIv3 to automatically generate indicators of compromise

TL;DR: VirusTotal APIv3 includes an endpoint to retrieve all the dynamic analysis reports for a given file. This article showcases programmatic retrieval of sandbox behaviour reports in order to produce indicators of compromise that you can use to power-up your network perimeter/endpoint defenses. We are also releasing a set of python scripts alongside this blog post to illustrate this use case.

We recently rolled out a new Windows dynamic analysis system called VirusTotal Jujubox. This new sandbox represents a major revamp of VirusTotal’s in-house behaviour analysis capabilities as well as a key addition to the multi-sandbox project, which already aggregates behaviour reports from more than 10 partners and the most popular operating systems.

Behaviour reports are often perceived as a mechanism to understand what an individual sample does when executed, a quick overview before diving into disassembly and debugging. However, when you have a massive dynamic analysis setup processing hundreds of thousands of files per day, the microscopic dissection capability is far from being the most attractive use case.

When you generate reports at scale, and more importantly, when you index them in an elasticsearch index and expose it via API, the generated data can be used for advanced hunting, especially when this data can be combined with other static, binary and in-the-wild properties.

The basic workflow would be as follows:

  1. Periodically identify new malware variants pertaining to a family that you are tracking making use of the VT Intelligence search API. Use family variant commonalities (for instance a section name, the compilation timestamp or a document’s author metadata property) to retrieve a stream of malware.
  2. Focus on recent matches since the previous execution (query: fs:2019-11-01+).
  3. For each match, retrieve the generated behaviour reports for the pertinent file. You can also focus specifically on network communications with the contacted_ips, contacted_domains and contacted_urls relationships.
  4. For each automatically extracted network observable, check popularity ranks in order to filter out noise and FPs.
  5. All the newly yielded network artefacts (CnCs) can then be fed into SIEMs or transformed into IDS rules to power up network perimeter defenses.

Let’s illustrate this with a particular example. Bankbot is an Android banking trojan, it allows the attacker to perform:

  • SMS hijacking.
  • GPS tracking.
  • New permission requests.
  • Overlay attacks to mask legit bank apps with forms to intercept credentials. Sometimes based on a remote set of HTML templates. 

The trojan was released in an underground forum and the post included the source code for the client-side and server-side components, including the database setup to collect stolen information.




Initially, the trojan included a hardcoded list of target bank applications that it would overlay in order to intercept banking credentials:



Since the source code of the trojan was also published in the underground forum, other crooks soon modified it to accept a remote list of financial entities to attack. This makes target identification more complex, static analysis is not enough to identify the targeted banks and subsequent date-tied CnC infrastructure.



While identifying targeted financial institutions might be a more complex task, discovering new variants of the same family and automatically identifying new network infrastructure tied to it becomes easier. Why is this? A server-side remote target list leads to a common network infrastructure pattern that can be used to track the malware family.

This is an example of a Bankbot sample:
https://www.virustotal.com/gui/file/5fdbe1e83ec9c43929cc348681cb6afde12afee637feaf444a4983c317b18423/detection

VT Enterprise allows similarity searches and other attribute searches to find additional variants of the same malware family. In this particular case, the Android package name under the details tab seems interesting, clicking on it will launch a VT Intelligence search for other Android APKs that share that very same package name:



The matches do indeed seem to belong to the same family:




When opening these samples and looking at their behaviour reports, certain commonalities are easily noticed:





Static/behaviour/code commonalities are very frequent since attackers usually reuse code across different campaigns. Sometimes the commonalities are a result of recompiling the same code to communicate with a different network infrastructure. Other times, commonalities are present because the attack binaries are generated with some kind of builder or kit for dummies. Similarly, CnC infrastructure often exhibits commonalities in terms of the same path structure or query parameters, it is the result of attackers reusing the same CnC panel through a server-side kit that they deploy without changing file names or path structure.

These patterns, in conjunction with VT’s massive dynamic analysis setup and indexing, make it easy to automatically discover new malicious network infrastructure and automatically generate indicators of compromise.

The behaviour reports for the identified cluster of samples shows that the CnC panel uses the subpaths tuk_tuk.php or checkPanel.php.

Let’s use this common pattern to periodically check VirusTotal for new variants of this malware family, and by doing so, let’s identify new network infrastructure tied to this attack, live, as samples are uploaded to VirusTotal.

Using the APIv3 Intelligence search endpoint, it’s possible to search for any Android APK whose network recordings contain the substring tuk_tuk.php:
https://developers.virustotal.com/v3.0/reference#intelligence-search
type:apk behaviour_network:"tuk_tuk.php"

Multiple properties, such as dynamic/static analysis and metadata, can be combined to make a more refined search:
type:apk behaviour_network:"tuk_tuk.php" behaviour:"del_sws" androguard:"android.permission.ACCESS_FINE_LOCATION"

The API can sort matches according to first seen descending, meaning that by executing this search periodically and focusing on the latest results, it’s possible to discover new malicious network infrastructure tied to this particular family.

At the time of writing, this search yielded the following results:

5fdbe1e83ec9c43929cc348681cb6afde12afee637feaf444a4983c317b18423
elis[.]ru
92.53.97[.]75
hxxp://elis.ru/private/tuk_tuk.php

52998a07d22b0aa267505635898219ef6104dc6cd255bea69c7ab701285666fa
xzcxzfs.kl.com[.]ua
5.79.66[.]145
hxxp://xzcxzfs.kl.com[.]ua/private/tuk_tuk.php

7c06552f59b594ef0d650204423e97c8ab8f07588f1215ec2a469dc9cb7f5670
u36084.test93w[.]ru
hxxp://u36084.test93w[.]ru/private/tuk_tuk.php

56b220e610d17987b4f96afa79e23c3c9cab16592384ed883e9ac8240907b53b
u36206.test93w[.]ru
185.31.163[.]148
hxxp://u36206.test93w[.]ru/private/tuk_tuk.php

The Intelligence search API endpoint will return a list of file objects matching a search criteria. Each of these file objects can have one or more multi-sandbox reports. These behaviour reports can be retrieved making use of the pertinent relationship (behaviours) for each of the files:
https://developers.virustotal.com/v3.0/reference#files-relationships

It’s also possible to filter the network communication relationships fields, instead of asking for the whole report (contacted_urls, contacted_ips, contacted_domains):
https://www.virustotal.com/api/v3/intelligence/search?query=type:apk behaviour_network:”tuk_tuk.php”&relationships=contacted_urls,contacted_domains,contacted_ips

Once the pertinent network infrastructure is parsed, it’s possible to either rely on the objects returned by the network-related relationships (contacted_urls, contacted_ips, contacted_domains) or make a subsequent automated call to the domain / IP address / URL API endpoint in order to retrieve further details about the given network observable. The aim of this subsequent stage is to filter out potential false positives. For instance, among the details returned for a domain lookup, there are different popularity rank lists that can be useful to filter out TOP domains.

You can easily test this workflow with a little script released along with this blog post. This script makes use of our official APIv3 python library, it can serve as your starting point to build more complex pipelines:
https://github.com/VirusTotal/vt-py/blob/master/examples/intelligence_search_to_network_infrastructure.py

python3 intelligence_search_to_network_infrastructure.py --apikey=<YOUR_API_KEY> --query=’type:apk behaviour_network:"tuk_tuk.php"’

=== Results: ===
DOMAIN: u363571.test93w.ru
IP_ADDRESS: 185.31.163.148
URL: http://u363571.test93w.ru/private/tuk_tuk.php
DOMAIN: bot.mymaster-rem.ru
URL: http://bot.mymaster-rem.ru/private/tuk_tuk.php
DOMAIN: lensfor.xyz
URL: https://lensfor.xyz/private/tuk_tuk.php
IP_ADDRESS: 38.21.243.204
URL: http://38.21.243.204/anib/private/tuk_tuk.php
DOMAIN: f0316480.xsph.ru
IP_ADDRESS: 141.8.192.151
URL: http://f0316480.xsph.ru/private/tuk_tuk.php
DOMAIN: u36255.test93w.ru
DOMAIN: mtalk4.google.com
IP_ADDRESS: 185.31.163.148
URL: http://u36255.test93w.ru/private/tuk_tuk.php
DOMAIN: u36206.test93w.ru
IP_ADDRESS: 185.31.163.148
URL: http://u36206.test93w.ru/private/tuk_tuk.php
DOMAIN: yumishop.co.uk
URL: http://yumishop.co.uk/private/inj_lst.php
URL: http://yumishop.co.uk/private/tuk_tuk.php
DOMAIN: u36317.test93w.ru
IP_ADDRESS: 185.31.163.148
URL: http://u36317.test93w.ru/private/tuk_tuk.php
DOMAIN: u36317.test93w.ru
IP_ADDRESS: 185.31.163.148
URL: http://u36317.test93w.ru/private/tuk_tuk.php


Note that this workflow is exclusively based on behavioural observations and works independently of the detection ratio of files, by pipelining VT Intelligence searches and sandbox report lookups, it is possible to generate indicators of compromise even if the related sample is undetected. The identified domains can be automatically checked against SIEM logs or can be automatically transformed into IDS rules, serving as an additional layer in your onion-like security strategy.

This blog post focuses on combining VT Intelligence searches with behaviour lookups, the same can be done with YARA rule matches. VT Hunting Livehunt matches can programmatically retrieved using APIv3, for each match the pertinent behaviour reports can be retrieved and CnC network infrastructure can be automatically extracted. Similarly, other properties that can be used as IoCs, such as mutexes, registry keys, embedded domains, file names, cmd parameters and the like can be automatically yielded. The following two script showcase this other VT Hunting workflow:
https://github.com/VirusTotal/vt-py/blob/master/examples/hunting_notifications_to_network_infrastructure.py
https://github.com/VirusTotal/vt-py/blob/master/examples/retrohunt_to_network_infrastructure.py

If you are rather a golang fan, feel free to check out our official VirusTotal golang library:
https://github.com/VirusTotal/vt-go

APIv3 was a major component of our 2019 roadmap, soon we will be officially releasing it and announcing a generous deprecation timeline for APIv2, stay tuned!

Wednesday, June 11, 2014

, , , , ,

VirusTotal API implementation in C programming language

Many users interact programmatically with VirusTotal via its public API, it is an easy HTTP+JSON interface that allows you to easily submit and check files in order to help improve security world-wide. Moreover, many VirusTotal Community volunteers have very kindly implemented the API in a wide variety of programming languages, some of these implementations are documented here, many others exist and we will progressively adding all those that we are made aware of.

This said, there was not any full implementation of the API in the C language, so that any C or C++ program that users might be building could easily interact with VirusTotal, at least we were not aware of any. We have released a VirusTotal interface written in C to our API  on github at https://github.com/VirusTotal/c-vtapi, any C or C++ program should be able to use it. Its goal is to implement all of the public API and private API features in C. The public API features will work for anyone with a free public API key, the private API features will only work for those who have licensed our services and use a private API key.

The recently announced VirusTotal Uploader for OS X internally uses the c-vtapi project. Using C it is a common building block that other programs or languages can interface to.

Suggestions, comments, patches and github pull request for improvements are welcome. Some ideas of improvements:
  • Better windows support and testing. We have tested a lot with OS X and linux, the windows scaffolding is there, but is not well tested.
  • More example programs or command line utilities that use this C API interface. For example, we know Sebastian Poeplau, being a busy guy, was looking for collaborators that would implement VirusTotal submissions in his awesome Ghost USB project, perhaps this C implementation makes it easier to perform the integration and some volunteers stand up.

Thursday, December 13, 2012

, , , ,

Public API request rate limits and tool development

Our goal is simple: to help keep you safe on the web. For this to happen, among many other technical fireworks, we need to receive as many (hopefully malicious) files as possible that we can eventually share with the antivirus and security industry in order to allow them to improve their products and technologies.

One of the ways we envisioned increased submissions to VirusTotal was through the release of our public API. Many tools and security deployments (honeypots, honeyclients, sandboxes, etc.) are making use of it and we are delighted that they do so. However, very often I see that integration with VirusTotal's API could be simpler.

Many users ignore the fact that public API request rate limits are enforced on (IP address, API key) tuples. What does this mean? Users sharing a same API key with different IP addresses will be subjected to independent request rate counters. Putting it simpler, if you are a tool developer, you might want to create a public API key for your tool and embed it in your application, that way, by default, you would not have to ask the user to create an API key and the whole integration with VirusTotal would be transparent. 

Having said this, it is always wise to have a settings file or tab that allows users to change this default key:
  • Some users might be behind some sort of proxy, corporate network aggregator, NATting device, or similar setup that makes them share the same IP address with other users of your tool, these users should be given the option to create their own API key and modify the setting in your tool.
  • Some users might just want to use an independent key in order to track their own submissions in their VirusTotal Community profile.
  • Some users might simply find the public API request rate limit too low, they might want to speak with VirusTotal about the possibility of getting a private API key, they should be able to embed that independent private API key in your setup.

So, imagine this hypothetical situation: I want to build a tool that whenever a USB storage device is plugged into a given PC it inspects its files, looks for any autorun.inf file and submits to VirusTotal any referenced executables in it. I would create an VirusTotal Community account for my tool and retrieve the corresponding API key, I would hardcode that into my application and make the tool use it by default. This said, I would also have a settings tab in my application that would allow users to change this key for any other key they might register. Of course, I would plan to render the corresponding messages informing a user about the fact they can modify the default key whenever request rate limitations are met because of IP sharing.

Hope this is useful and I would love to see more VirusTotal plugins out there with a more transparent integration such as the one described above. As usual, before doing any kind of integration please look at our Terms of Service and Best practices, tools competing with the antivirus industry or jeopardizing such industry will be immediately banned from the service. VirusTotal is a tool to help antivirus vendors in improving their products, not a means to discredit, harm them in any way or steal their intellectual property, we take this matter very seriously.