Deep Dive into AMP and Threat Grid integration with Cisco Email Security

In this blog post, we are going to dive deeper and explain the workflows of AMP and Threat Grid integration with Cisco Email Security (applies to both Cloud Email Security and on premise Email Security Appliance), as well as help administrators refine security posture in their organizations. Let’s start with a quick recap of how file reputation, file analysis and file retrospection work together in general.

File Reputation service allows the ability capture a file on a network, email, web gateway or on the endpoint, calculate a hash and query the AMP cloud to receive a disposition back – either clean, malicious or unknown. Malicious and clean files are normally not a subject for additional investigations and a policy action can be taken accordingly. For unknown files, this is when we want to provide additional analysis – we can do so by taking the file out of the network and uploading it up to the File Analysis service – Threat Grid. Threat Grid applies both static and dynamic analysis techniques and records results of file execution into a human-readable analysis report. It also issues a threat score overall. The two together help determine how likely it is that the file is malicious. The AMP cloud may be updated with the analysis results from Threat Grid, which can lead to AMP cloud changing the disposition for a given file. Cisco Talos also constantly pushes intelligence about the files they analyze into the AMP cloud, which complements AMP’s global intelligence. This can trigger retrospective events, that help us notify our customers about all the locations where these files were seen on their network – whether it was seen by network or content gateway or the endpoint, depending on where you have deployed the AMP license. What’s important to remember is that the authoritative source to convict a file is the AMP cloud, not Threat Grid.

Now let’s have a look at how we can apply the concepts, that we have just reviewed to Cisco Email Security solution (referred to as “ESA” further). AMP is a name of an add-on license for Cisco ESA, which brings:

• capability to run file reputation queries on attachments against the AMP cloud
• capability to submit unknown attachments that meet the criteria to Threat Grid
• receive retrospective notifications from AMP, in case of a disposition change

So, where do those capabilities sit it in the ESA workqueue?

Assuming the message wasn’t blocked by the preceding ESA inspection layers, such as sender reputation, message filters, multiple anti-spam engines, multiple anti-virus engines – the message arrives to AMP and Threat Grid inspection point.

AMP File Reputation Workflow

In the first phase, ESA attempts to derive the disposition of the attachment from AMP, let’s break it down and review the exact steps taken by ESA in this phase.

When a message with an attachment reaches AMP after anti-virus scanning, ESA attempts to parse the attachment from the message by checking the message headers (check for compliance with RFC 2045). Even if the message is not fully compliant, ESA still makes best effort to parse the attachment. The next step is to check whether an attachment is an archive file and if so – attempt to unpack it. If any of the above steps fail, due to for example format errors or file corruption, the configurable policy for unscannable attachments comes into effect.

The files (along with the original compressed archive, if applicable) are then sent to the next step – checking of the internal ESA AMP cache to understand whether a disposition of this file was already queried in the past and whether it could be now derived from cache. On a side note, a useful addition in ESA 11.0 is the ability to configure the file reputation cache time to live, giving administrators more granular control over the cache usage. If the cache doesn’t contain an entry for this file, ESA will communicate with the AMP Cloud (public or private) to query the file reputation, which will return back a verdict: either clean, malicious or unknown. Clean files continue through ESA workqueue to perform graymail detection, content filtering and outbreak filtering inspections, if configured to do so. Malicious files are processed according to the configured policy. It’s important to keep in mind that if an archive has multiple files inside – if even one is malicious then the entire archive and message will be seen as malicious. Attachments with unknown disposition are treated differently and they may be requested by the AMP Cloud for upload to Threat Grid – this may happen when file analysis results for a given attachment are not available in the AMP cloud, meaning they were not shared by Threat Grid in the past, likely because the attachment was not analysed in Threat Grid. Such files can proceed to the next phase. 

File Upload Criteria Workflow

In the second phase, ESA performs a couple of checks to see if the unknown file meets the upload criteria and if it contains suspicious content, that could likely show up as malicious.

ESA first checks whether a file meets the following criteria:

• supported file type – at the time of File Analysis configuration, ESA administrator can select the desired file types
• does not exceed the file size threshold defined by Threat Grid

If the two criteria above are met, the attachment continues to the next step – ClamAV pre-classification check. This step helps determine whether there is dynamic content and object streams inside, such as macros, embedded EXE, flash, etc. This step is needed to ensure that only files that can possibly be malicious are uploaded to Threat Grid, and others that have no chance of being malicious are not uploaded and do not burn out file upload limits unnecessarily.

If either of those criteria are not met – the message continues through the workqueue without uploading the file to Threat Grid. Alternatively, if both criteria are met, ESA proceeds to the next phase.

Threat Grid File Analysis Workflow

In the third phase, more validations are performed before ESA finally uploads the attachment for analysis to Threat Grid. Let’s have a look at the workflow.

In this phase, the first couple of steps for ESA are to check whether the local file upload queue is full or not and whether Threat Grid (public or appliance) is reachable. If either of these conditions is not met, the attachment is not sent for analysis and the message continues through ESA workqueue (Content Filters and Outbreak Filters). Assuming the local upload queue is not full and Threat Grid is reachable, ESA proceeds by placing the associated message into File Analysis quarantine and by checking whether the attachment was already uploaded to Threat Grid by another device (for example, another ESA). If that’s the case, a duplicate will not be uploaded for analysis again. Alternatively, if the attachment is not yet known to Threat Grid, ESA would proceed and submit the file for analysis. This time it’s up to Threat Grid to check if the sample upload limit was reached. If that’s the case, Threat Grid discards the request and the associated message is released from quarantine. Customers can easily add more daily sample submissions to Threat Grid through Sample Packs or Premium subscription.

If the upload limit wasn’t reached, the file gets accepted and queued by Threat Grid. Simultaneously, ESA adds a record of the SHA256 of this file to its internal database (where it’s kept for up to 12 hours) and starts periodically querying if the analysis was complete, until it receives a positive response back from the File Analysis service. If there is no “file analysis complete” message from Threat Grid within 12 hours and if the File Analysis quarantine was configured to hold the message that long, the SHA256 ages out and ESA releases the message from quarantine to the workqueue. Alternatively, once Threat Grid analysis is complete, the results of this analysis are added to the AMP cache on ESA. At the same time, Threat Grid shares this information with the AMP cloud, so that other AMP and Threat Grid integrated devices on the network can take advantage of the new intelligence. Threat Grid cloud can share analysis results with AMP public cloud and Threat Grid appliance can share results with AMP private cloud, but not the other way around.

Along with the AMP cache update on ESA and the intelligence sharing between Threat Grid and AMP clouds, the associated message with an attachment is released from File Analysis quarantine. Further workqueue rescanning would skip the File Analysis workflow, since File Reputation query would use the updated AMP cache to derive a disposition for this file. Even if ESA still derives an ‘unknown’ disposition from cache, the upload to File Analysis service wouldn’t happen again, since the file is already known to Threat Grid.

It’s important to keep in mind, that either ESA or the AMP cloud can convict a file based on a threat score returned back after analysis. Threat Grid itself is not a solution to convict files or assign disposition to files directly, that’s also one of the reasons customers sometimes would see significant numbers of unknowns.

Retrospective Verdict

File verdicts can change as new information emerges – we’ve mentioned that AMP cloud can change file dispositions based on Talos analysis and based on Threat Grid analysis. Cisco ESA is constantly staying in touch with the AMP cloud by sending a periodic Heartbeat message, which also asks the cloud if there were any changes in dispositions of files, that were sent through ESA. If there was indeed a disposition change for a particular file that passed through AMP and Threat Grid inspection on ESA, the solution would alert the administrator, specifying the details necessary to go back and perform proper investigation. A notification includes information about the message and the attachment – such as subject, sender and recipient, file name and hash, and a new disposition.

The best way to track down how AMP and Threat Grid inspection works on your Cisco Email Security solution is to review the reports presented in the user interface, as well as to follow the traces in the AMP Engine logs. This information combined together will present a clear idea to the Email Security administrators about how File Reputation and File Analysis services work.

Securing your organization from advanced email-based threats is not an easy task and requires a multi-layered approach with all the inspection layers tightly working together and complementing each other. Make sure to always include Threat Grid Premium subscription with your AMP on Cisco Email Security evaluations to get access to Threat Grid cloud portal for manual file and URL uploads, extensive reporting, API for further integrations and premium threat intelligence feeds