Blog

Archive for June 17th, 2021

Network Forensics on Azure VMs (Part #2), (Fri, Jun 18th)

In yesterday’s diary, we took a look at two methods that allow to capture network connection information off a potentially compromised virtual machine in Azure. Today, we’ll investigate the most recent addition to the VM monitoring arsenal, namely “Azure Monitor Insights”.

“Insights” is enabled directly under the “Monitoring” menu tab of the corresponding VM. Deploying it can be done from within the Azure Portal, while a VM is running, and without having to log in on the VM itself. The solution deploys a Microsoft OMS monitoring agent into the VM though, so this isn’t exactly stealthy either.

Unlike the two methods shown in yesterday’s diary, “Insights” combines process telemetry from within the VM with network flow logs. The resulting charts are meant well, but get unwieldy very quickly. Behind the charts, there is though a lot of data that can be reached via click-through:

 

In this case, we can see that the process “wget” made connections on Port 80 and 443, and in the details pane, we can even see the start time, working directory, and the command line used.

 

But wait, there’s more. The “Insights” chart panel is just visualizing information that is also directly accessible, in the associated Azure Log Analytics container. With the right query in Kusto Query Language (KQL), we can search, combine, merge and dice directly on the logs themselves. This allows for example to quickly identify which process (if any) is leaking or uploading large volumes of data, and to where:

When you experiment with Insights for the first time, keep an eye on the related costs. The pricing model of Azure Monitor Insights is a bit unpredictable, and depends on the volume stored in the associated Log Analytics container. If you have a busy machine that generates a lot of log data, the “free” 5GB allotment in the current Pay-as-you-go pricing model can be used up quite quickly. See https://azure.microsoft.com/en-us/pricing/details/monitor/ for details.

If you have additional tips on how to conduct forensic network monitoring on Azure VMs, please let us know, or share in the comments below.

 

 

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →

Network Forensics on Azure VMs (Part #1), (Thu, Jun 17th)

The tooling to investigate a potentially malicious event on an Azure Cloud VM is still in its infancy. We have covered before (Forensicating Azure VMs) how we can create a snapshot of the OS disk of a running VM. Snapshotting and then killing off the infected VM is very straight forward, but it also tips off an intruder that he has been found out. Sometimes, it makes sense to first watch for a while, and learn more, for example about compromised accounts, lateral movement, or other involved hosts.

And to that end … we would like to capture the network traffic to and from our affected VM.  In an “old-fashioned” on-premises datacenter, we would in such a situation likely deploy an RSPAN port to mirror the traffic to a collector. A similar functionality was once in the making for Azure with “Virtual Network Tap” but this solution has been put on ice by Microsoft, and is not currently available.

Of course we could also log into the VM and start Wireshark or tcpdump … but this wouldn’t be exactly subtle, and if the machine indeed were compromised, doing so would for sure tip off the attacker.

So … which options are there?

Option 1:  Network Watcher Packet Capture

This delivers a full PCAP, and Microsoft amazingly even managed not to “improve” the PCAP format. Which means that the PCAPs collected with Network Watcher can in fact be opened and analyzed Wireshark. I’m sure this was just an oversight on Microsoft’s part, and they are currently busy re-implementing packet capture to produce a JSON or Excel file instead 🙂

Joking aside, the main downside of this Network Watcher PCAP is that the capture duration is limited to 18000 seconds aka five hours. So you need to add some sort of logic to (or have an operator manually) restart the capture every couple hours, to make sure no data is lost. The second downside is that the PCAP is written to an Azure Blob Storage account, and the caching/timing of those writes is mightily unpredictable. If you expect that you can “watch live” what is going on on the VM, think again. The PCAPs show up with delays of half a hour or more.  

The main advantage is … well, it is full PCAPs, and packets rule! Another advantage is that Network Watcher PCAP can be deployed onto a VM interface from within the Azure Portal, without touching the VM itself, and while the VM keeps running. An intruder who has an eye on the processes within the VM would notice though – while the watcher gets deployed via the virtualization layer, it does result in running “Network Watcher” processes that are visible from within the VM.

To enable Network Watcher for your VM, search for “Network Watcher” from the main Azure Portal search bar. Enable the Watcher for the Azure region(s) where you have resources of interest. Then open Network Watcher, and click on “Packet Capture” in the menu on the left.

Option 2: Flow Logs on the Network Interface Security Group (NSG)

This is one of the older techniques. Flow logs don’t provide full packet captures, but create a log of source/destination IPs and ports, timestamp, and bytes transferred in each direction. Better than nothing. The main disadvantage is that in this case, Microsoft did reinvent the format, and it is a puke pile of JSON where, to their shame, even Microsoft’s own recommendation seems to be to import the logs into Elastic or Splunk, or – yikes – to evaluate them using PowerBI. In a pinch, the format can though also be parsed with a bit of Powershell. https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview explains the structure.

To enable Flow Logs, open the NSG attached to the interface or VNET in question. The option “NSG Flow Logs” in the menu on the left, under “Monitoring”, is where you enable the forwarding of flow logs to an Azure Storage Account.

In tomorrow’s diary, we’re going to look at a third option, using Azure Monitor Insights.

 

 

 

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →