The Power of Logging in Incident Response

A deep dive into logging as an often-overlooked but powerful tool for incident detection and response

“Lack of instrumentation or insufficient logging” is often a phrase used on incident response reports. During incident response activities, this isn’t a phrase you want to see, since lack of logging inhibits your organization’s ability to conclusively determine root cause analysis.

As a team leader on Cisco Security Incident Response Services team, I work with a wide-range of organizations, from small-medium businesses to some of the largest and most complex environments across the globe. Many organizations today still fail at adequate logging across the enterprise, impeding their ability to perform intelligent-driven incident response tasks. This is in spite of the fact that the importance of logging is something that is constantly communicated in blogs, talks, IR reports, and even boardrooms!  I’ll say it again: Log sources are critical to intelligent incident response.

Thankfully, there are free and cost-for sufficient logging on your endpoints. Remember, an attacker’s technique is only as sophisticated as it needs to be to carryout Actions on Objectives in your network. An attacker isn’t going to drop a 0-day[1] when there are many ways to gain a foothold in an organization’s network. PowerShell is an effective, secure way of managing your network’s Windows infrastructure. Unfortunately, PowerShell attacks are prevalent and a very effective way for an attacker to execute arbitrary code on your endpoints.

PowerShell Logging

Why PowerShell? PowerShell is native, supported across most versions of Windows, and offers full access to Windows Management Instrumentation (WMI). Finally, many organizations are not logging PowerShell, which means you have the ingredients for the perfect storm.

We recommend customers turn on Module Logging, Script Block Logging, and Transcription Logging via Group Policy Object (GPO). The Microsoft PowerShell team highlights these advantages and many security features for the Blue Team.

In the event Mimikatz[2] harvests credentials in your Windows environment, you’ll have a detection mechanism (+1: blue team), but also plaintext credentials and hashes likely dumped to your event logs (+1: red team, but -1: blue team). Without logging or an endpoint detection and response (EDR) solution in place, you won’t be able to detect a memory (RAM) only credential harvesting attack via PowerShell. As a defender, it is critical to understand and manage risk to capabilities and business requirements.

A centralized logging solution is highly recommended for PowerShell and Sysmon logging. In addition to centralized logging, you’ll also need to tune/manage your configuration and deployment due to the additional logging requirements needed to detect and respond to a PowerShell attack.

Sysmon

System Monitor (Sysmon) is another tool freely available from Microsoft. When our Cisco Security Incident Response Services team is working with organizations to develop a robust incident response plan, we’ll recommend PowerShell Logging and Sysmon capabilities as a baseline for any endpoint detection.

AMP for Endpoints

If you have an EDR solution like Cisco Advanced Malware Protection (AMP) for Endpoints, you have yet another mechanism for detecting a PowerShell attack. In a recent lab exercise, we tested our PowerShell attack on the victim workstation without AMP for Endpoints and a second test with AMP for Endpoints, which did block Mimikatz from harvesting credentials from our victim workstation during lab testing.

Within our Cisco AMP for Endpoints console, we enabled command line capture (Policies > Advanced Settings), Block (Network), and Exploit Prevention (see Figure 1 below), which did block Mimikatz from successfully harvesting our lab user’s credentials, confirmed through PowerShell Transcription logging.

Figure 1: AMP for Endpoint Console (Policies: Exploit Prevention)

From a rapid incident response perspective, two powerful features within the Cisco AMP for Endpoints console are Device Trajectory and File Trajectory. Device Trajectory provides visibility into events that occurred leading up to and following a compromise, including parent processes, connections to remote hosts, and unknown files that may have been download by malware. In Figure 2 below, we can see the Encoded PowerShell command indicator of compromise (IOC) that was blocked by AMP for Endpoints. Retrospectively, we can also see powershell.exe, which is benign. However, a seasoned incident responder understands temporal proximity and would hypothesize that a benign portable executable (powershell.exe) may have been used to pivot, bypass AV, and/or download malware.

Figure 2: AMP for Endpoints Console (Device Trajectory)

Based upon the logs (PowerShell and Sysmon), and our EDR solution (Cisco AMP for Endpoints), we would want to get a RAM dump of this compromised endpoint for forensic analysis to pursue root cause in our incident response investigation.

Memory Forensics

During the incident response process, memory forensic analysis of infected endpoints is a common activity for your incident response team. This capability is critical to detection and response. For this “fileless” (no binary written to disk) PowerShell attack where you lack logging and visibility, memory forensics is your only option on the endpoint. Do you have this capability?

During this exercise, I used Volatility (An Advanced Memory Forensics Framework) to perform memory forensic analysis to detect Invoke-Mimikatz PowerShell script running in memory. Using the Volatility’s yarascan plugin and the Mimikatz yara[3] rule (kiwi_passwords.yar), I executed the following command:

Below (Figure 3) is the command output snippet identifying the lsadump module of mimikatz running in svchost.exe:

Figure 3: YARA: Mimikatz Detection (lsadump rule)

In summary, PowerShell logging, Sysmon, an EDR solution such as Cisco AMP for Endpoints, and a memory forensics capability are vital processes to efficient incident response. This multi-layered approach allows for detection and response, but more importantly if one capability fails (i.e. event logs are overwritten, due to size, cleared by an attacker, etc.) you have another mechanism to detect (i.e. PowerShell logging, EDR solution, Memory Forensic analysis) a common PowerShell attack.

If you have an in-house capability, fantastic! Definitely make sure you are using logging to its full potential.  If you are looking to augment your incident response capabilities, please consider leveraging our team. We stand ready to assist your organization maneuver treacherous waters during an incident and also those calm seas with our proactive Incident Response services portfolio.

[1] Zero-Day Vulnerability: https://en.wikipedia.org/wiki/Zero-day_(computing)

[2] Mimikatz: https://github.com/gentilkiwi/mimikatz

[3] YARA: http://yara.readthedocs.io/en/v3.7.0/index.html