NETRESEC Network Security Blog - Tag : tshark
Jason Reaves gave a talk titled “Malware C2 over x509 certificate exchange” at BSides Springfield 2017, where he demonstrated that the SSL handshake can be abused by malware as a covert command-and-control (C2) channel.
He got the idea while analyzing the Vawtrak malware after discovering that it read multiple fields in the X.509 certificate provided by the server before proceeding. Jason initially thought these fields were used as a C2 channel, but then realized that Vawtrak performed a variant of certificate pinning in order to discover SSL man-in-the-middle attempts.
Nevertheless, Jason decided to actually implement a proof-of-concept (PoC) that uses the X.509 certificate as a C2 channel. Jason’s code is now available on GitHub along with a PCAP file demonstrating this covert C2 channel. Of course I couldn’t resist having a little look at this PCAP file in NetworkMiner.
The first thing I noticed was that the proof-of-concept PCAP ran the SSL session on TCP 4433, which prevented NetworkMiner from parsing the traffic as SSL. However, I was able to parse the SSL traffic with NetworkMiner Professional just fine thanks to the port-independent-protocol-identification feature (a.k.a Dynamic Port Detection), which made the Pro-version parse TCP 4433 as SSL/TLS.
Image: X.509 certificates extracted from PCAP with NetworkMiner
A “normal” x509 certificate size is usually around 1kB, so certificates that are 11kB should be considered as anomalies. Also, opening one of these .cer files reveals an extremely large value in the Subject Key Identifier field.
Not only is this field very large, it also starts with the familiar “4D 5A” MZ header sequence.
NetworkMiner additionally parses details from the certificates that it extracts from PCAP files, so the Subject Key Identifier field is actually accessible from within NetworkMiner, as shown in the screenshot below.
You can also see that NetworkMiner validates the certificate using the local trusted root certificates.
Not surprisingly, this certificates is not trusted
Extracting the MZ Binary from the Covert X.509 Channel
Even though NetworkMiner excels at pulling out files from PCAPs, this is definitively an occasion where manual handling is required. Jason’s PoC implementation actually uses a whopping 79 individual certificates in order to transfer this Mimikatz binary, which is 785 kB.
Here’s a tshark oneliner you can use to extract the Mimikatz binary from Jason's example PCAP file.
tshark -r mimikatz_sent.pcap -Y 'ssl.handshake.certificate_length gt 2000' -T fields -e x509ce.SubjectKeyIdentifier -d tcp.port==4433,ssl | tr -d ':\n' | xxd -r -p > mimikatz.exe
Detecting x509 Anomalies
Even though covert channels using x509 certificates isn’t a “thing” (yet?) it’s still a good idea to think about how this type of covert signaling can be detected. Just looking for large Subject Key Identifier fields is probably too specific, since there are other fields and extensions in X.509 that could also be used to transmit data. A better approach would be to alert on certificates larger than, let’s say, 3kB. Multiple certificates can also be chained together in a single TLS handshake certificate record, so it would also make sense to look for handshake records larger than 8kB (rough estimate).
This type of anomaly-centric intrusion detection is typically best done using the Bro IDS, which provides easy programmatic access to the X.509 certificate and SSL handshake.
There will be false positives when alerting on large certificates in this manner, which is why I recommend to also check if the certificates have been signed by a trusted root or not. A certificate that is signed by a trusted root is very unlikely to contain malicious data.
Posted by Erik Hjelmvik on Tuesday, 06 February 2018 12:13:00 (UTC/GMT)
Remember the days back in the 90s when you could cripple someones Internet connection simply by issuing a few PING command like “ping -t [target]”? This type of attack was only successful if the victim was on a dial-up modem connection. However, it turns out that a similar form of ICMP flooding can still be used to perform a denial of service attack; even when the victim is on a gigabit network.
The 90's called and wanted their ICMP flood attack back
Analysts at TDC-SOC-CERT (Security Operations Center of the Danish telecom operator TDC) noticed how a certain type of distributed denial-of-service (DDoS) attacks were more effective than others. The analysts found that a special type of ICMP flooding attack could disrupt the network throughput for some customers, even if the attack was just using a modest bandwidth (less than 20Mbit/s). It turned out that Destination Unreachable ICMP messages (ICMP type 3), such as “port unreachable” (code 3) was consuming significantly more resources on some firewalls compared to the more common ICMP Echo messages associated with the Ping command. The TDC team have dubbed this particular ICMP flooding attack method “BlackNurse”.
TDC's own report about BlackNurse says:
“The BlackNurse attack attracted our attention, because in our anti-DDoS solution we experienced that even though traffic speed and packets per second were very low, this attack could keep our customers' operations down. This even applied to customers with large internet uplinks and large enterprise firewalls in place.”
Cisco ASA firewalls is one product line that can be flooded using the BlackNurse attack. Cisco were informed about the BlackNurse attack in June this year, but they decided to not classify this vulnerability as a security issue. Because of this there is no CVE or other vulnerability number associated with BlackNurse.
Evaluation of BlackNurse Denial-of-Service Attacks
Members of the TDC-SOC-CERT set up a lab network to evaluate how effective ICMP type 3 attacks were compared to other ICMP flooding methods. In this setup they used hping3 to send ICMP floods like this:
ICMP net unreachable (ICMP type 3, code 0):
hping3 --icmp -C 3 -K 0 --flood [target]
ICMP port unreachable (ICMP type 3, code 3) a.k.a. “BlackNurse”:
hping3 --icmp -C 3 -K 3 --flood [target]
ICMP Echo (Ping):
hping3 --icmp -C 8 -K 0 --flood [target]
ICMP Echo with code 3:
hping3 --icmp -C 8 -K 3 --flood [target]
The tests showed that Cisco ASA devices used more CPU resources to process the destination unreachable flood attacks (type 3) compared to the ICMP Echo traffic. As a result of this the firewalls start dropping packets, which should otherwise have been forwarded by the firewall, when hit by a BlackNurse attack. When the packet drops become significant the customer behind the firewall basically drops off the internet.
The tests also showed that a single attacking machine running hping3 could, on its own, produce enough ICMP type 3 code 3 packets to consume pretty much all the firewall's resources. Members of the TDC-SOC-CERT shared a few PCAP files from their tests with me, so that their results could be verified. One set of these PCAP files contained only the attack traffic, where the first part was generated using the following command:
hping3 --icmp -C 3 -K 3 -i u200 [target]
The “-i u200” in the command above instructs hping3 to send one packet every 200 microseconds. This packet rate can be verified simply by reading the PCAP file with a command like this:
tshark -c 10 -r attack_record_00001.pcapng -T fields -e frame.time_relative -e frame.time_delta -e frame.len -e icmp.type -e icmp.code
0.000000000 0.000000000 72 3 3
0.000207000 0.000207000 72 3 3
0.000415000 0.000208000 72 3 3
0.000623000 0.000208000 72 3 3
0.000830000 0.000207000 72 3 3
0.001038000 0.000208000 72 3 3
0.001246000 0.000208000 72 3 3
0.001454000 0.000208000 72 3 3
0.001661000 0.000207000 72 3 3
0.001869000 0.000208000 72 3 3
The tshark output confirms that hping3 sent an ICMP type 3 code 3 (a.k.a. “port unreachable”) packet every 208 microseconds, which amounts to rougly 5000 packets per second (pps) or 2.7 Mbit/s. We can also use the capinfos tool from the wireshark/tshark suite to confirm the packet rate and bandwidth like this:
Number of packets: 48 k
File size: 5000 kB
Data size: 3461 kB
Capture duration: 9.999656 seconds
First packet time: 2016-06-08 12:25:19.811508
Last packet time: 2016-06-08 12:25:29.811164
Data byte rate: 346 kBps
Data bit rate: 2769 kbps
Average packet size: 72.00 bytes
Average packet rate: 4808 packets/s
A few minutes later they upped the packet rate, by using the “--flood” argument, instead of the 200 microsecond inter-packet delay, like this:
hping3 --icmp -C 3 -K 3 --flood [target]
Number of packets: 3037 k
File size: 315 MB
Data size: 218 MB
Capture duration: 9.999996 seconds
First packet time: 2016-06-08 12:26:19.811324
Last packet time: 2016-06-08 12:26:29.811320
Data byte rate: 21 MBps
Data bit rate: 174 Mbps
Average packet size: 72.00 bytes
Average packet rate: 303 kpackets/s
The capinfos output reveals that hping3 was able to push a whopping 303.000 packets per second (174 Mbit/s), which is way more than what is needed to overload a network device vulnerable to the BlackNurse attack. Unfortunately the PCAP files I got did not contain enough normal Internet background traffic to reliably measure the degradation of the throughput during the denial of service attack, so I had to resort to alternative methods. The approach I found most useful for detecting disruptions in the network traffic was to look at the roundtrip times of TCP packets over time.
The graph above measures the time between a TCP data packet and the ACK response of that data segment (called “tcp.analysis.ack_rtt” in Wireshark). The graph shows that the round trip time only rippled a little due to the 5000 pps BlackNurse attack, but then skyrocketed as a result of the 303 kpps flood. This essentially means that “normal” traffic was was prevented from getting though the firewall until the 303 kpps ICMP flood was stopped. However, also notice that even a sustained attack of just 37 kpps (21 Mbit/s or 27 μs inter-packet delay) can be enough to take a gigabit firewall offline.
Detecting BlackNurse Attacks
TDC-SOC-CERT have released the following SNORT IDS rules for detecting the BlackNurse attack:
alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"TDC-SOC - Possible BlackNurse attack from external source "; itype:3; icode:3; detection_filter:track by_dst, count 250, seconds 1; reference:url, soc.tdc.dk/blacknurse/blacknurse.pdf; metadata:TDC-SOC-CERT,18032016; priority:3; sid:88000012; rev:1;)
alert icmp $HOME_NET any -> $EXTERNAL_NET any (msg:"TDC-SOC - Possible BlackNurse attack from internal source"; itype:3; icode:3; detection_filter:track by_dst, count 250, seconds 1; reference:url, soc.tdc.dk/blacknurse/blacknurse.pdf; metadata:TDC-SOC-CERT,18032016; priority:3; sid:88000013; rev:1;)
Protecting against BlackNurse Attacks
The recommendation from TDC is to deny ICMP type 3 messages sent to the WAN interface of Cisco ASA firewalls in order to prevent the BlackNurse attack. However, before doing so, please read the following excerpt from the Cisco ASA 5500 Series Command Reference:
“We recommend that you grant permission for the ICMP unreachable message type (type 3). Denying ICMP unreachable messages disables ICMP Path MTU discovery, which can halt IPSec and PPTP traffic. See RFC 1195 and RFC 1435 for details about Path MTU Discovery.”
In order to allow Path MTU discovery to function you will need to allow at least ICMP type 3 code 4 packets (fragmentation needed) to be received by the firewall. Unfortunately filtering or rate-limiting on a Cisco ASA does not seem to have an affect against the BlackNurse attack, the CPU max out anyway. Our best recommendation for protecting a Cisco ASA firewall against the BlackNurse attack is therefore to rate-limit incoming ICMP traffic on an upstream router.
Another alternative is to upgrade the Cisco ASA to a more high-end one with multiple CPU cores, since the BlackNurse attack seems to not be as effective on muti-core ASA's. A third mitigation option is to use a firewall from a different vendor than Cisco. However, please note that it's likely that other vendors also have products that are vulnerable to the BlackNurse attack.
Update November 12, 2016
Devices verified by TDC to be vulnerable to the BlackNurse attack:
- Cisco ASA 5505, 5506, 5515, 5525 and 5540 (default settings)
- Cisco ASA 5550 (Legacy) and 5515-X (latest generation)
- Cisco 897 router
- Cisco 6500 router (with SUP2T and Netflow v9 on the inbound interface)
- Fortigate 60c and 100D (even with drop ICMP on). See response from Fortinet.
- Fortinet v5.4.1 (one CPU consumed)
- Palo Alto (unless ICMP Flood DoS protection is activated). See advisory from Palo Alto.
- SonicWall (if misconfigured)
- Zyxel NWA3560-N (wireless attack from LAN Side)
- Zyxel Zywall USG50
Update November 17, 2016
There seems to be some confusion/amusement/discussion going on regarding why this attack is called the “BlackNurse”. Also, googling “black nurse” might not be 100 percent safe-for-work, since you risk getting search results with inappropriate videos that have nothing to do with this attack.
The term “BlackNurse”, which has been used within the TDC SOC for some time to denote the “ICMP 3,3” attack, is actually referring to the two guys at the SOC who noticed how surprisingly effective this attack was. One of these guys is a former blacksmith and the other a nurse, which was why a college of theirs jokingly came up with the name “BlackNurse”. However, although it was first intended as a joke, the team decided to call the attack “BlackNurse” even when going public about it.
Posted by Erik Hjelmvik on Thursday, 10 November 2016 07:40:00 (UTC/GMT)
Users of Wireshark 1.8.0 (or later) have most likely noticed that the default output file format has changed from libpcap (.pcap) to Pcap-NG (.pcapng). So what does this mean other than a longer file extension?
New Features in PcapNG
PcapNG is short for “PCAP Next Generation Dump File Format”. This new format isn't just an update of the old PCAP specification, but a total rewrite of the whole spec. Here are a few of the features that are available in the new PcapNG format:
- Traffic captured from multiple interfaces can be stored in a single file. This even works when the interfaces have different data link types, such as Ethernet, 802.11 (WiFi) and PPP.
- PcapNG trace files can be tagged with metadata info about what OS, hardware and sniffer application that was used to capture the traffic. Wireshark and dumpcap automatically tags generated PcapNG files this way.
- There is a long list of metadata attributes that can be stored about each interface. This list includes attributes like interface name, dropped packets and used capture filter.
- Text comments can be added and saved to individual frames. These comments, which are called “annotations”, are available in Wireshark and Tshark via the display filter named “pkt_comment”.
You can, for example, use tshark to list all annotations in a PcapNG file with the following command:
tshark -r dump.pcapng -T fields -e pkt_comment -R pkt_comment
You might also have seen improved timestamp resolution as a new feature of the PcapNG format. It is true that the PcapNG format allows for more precise timestamps compared to the microsecond resolution provided by the old PCAP format. This does, however, not mean that you will get nanosecond resolution in your capture files just because you switch to the PcapNG format. The default timestamp resolution in PcapNG files is still microseconds, and tools like Wireshark and dumpcap additionally only get microsecond resolution timestamps from the library they rely on for packet capturing (i.e. libpcap or WinPcap).
Many tools are not yet able to load PcapNG files, instead they'll spit out error messages like “bad dump file format”.
These error messages typically appear when a tool parses PCAP files with help of a libpcap version
prior to 1.1.0 (you can find the error message in
savefile.c from early 2010).
If you instead load a PcapNG file into NetworkMiner you'll currently get the following error message:
NetworkMiner 1.4.1 with error message while trying to open a PcapNG file
Convert PcapNG to PCAP
If you end up with a capture file in the PcapNG format, which you need to parse with a tool that does not yet support the “NG” format, then you'll have to convert it to the legacy PCAP format. You can use editcap to do the conversion like this:
editcap -F libpcap dump.pcapng dump.pcapHowever, if the PcapNG file contains traffic from multiple interfaces with different data link types then you'll get an error message saying:
editcap: Can't open or create dump.pcap: That file format doesn't support per-packet encapsulations
One way to convert such multi-encapsulation PcapNG files back to PCAP format is to first split the capture file into multiple PcapNG files, by using the display filters “frame.interface_id” or “frame.dlt”, and then convert each individual PcapNG file back to PCAP using editcap.
Another option is to submit your PcapNG file to the conversion service provided at pcapng.com. This free service only lets you convert file sizes up to 1MB, but it's really easy to use and you don't need to sign up or register to use it. Just choose a file and press “Convert to PCAP”, the site will then provide you with PCAP formatted capture files for each interface.
Sniff to PCAP
If you want to avoid conversion issues altogether then it's probably safest to stay with the good ol' PCAP format. But since dumpcap defaults to PcapNG nowadays you'll have to tell it to use the legacy PCAP format with the “-P” switch like this:
dumpcap -P -i eth0 -w dump.pcap
Posted by Erik Hjelmvik on Wednesday, 05 December 2012 12:15:00 (UTC/GMT)
A customer recently contacted us because he wanted to load ERF capture files from their Endace probes into NetworkMiner Professional. In order to do so they would first need to convert the ERF file into the libpcap format. The obvious solution is to use editcap and specify the output capture type with “-F libpcap”. However, it turns out that by doing so the captured data in each frame will still be preceded by an ERF header. This causes a problem for tools that handle pcap files but do not have a parser for the ERF header format.
In order to convert capture files from the ERF format into PCAP without any ERF headers you need to specify not only the capture type but also the encapsulation type of the data inside the ERF headers. One way to see what encapsulation type to use is to run the Protocol Hierarchy Statistics (PHS) function in tshark. Here is what the PHS look like on the publicly available capture file erf-ethernet-example.erf:
tshark -r erf-ethernet-example.erf -q -z io,phs
Protocol Hierarchy Statistics
erf frames:19 bytes:7269 eth frames:19 bytes:7269 ip frames:19 bytes:7269 tcp frames:19 bytes:7269 http frames:4 bytes:2077 image-gif frames:2 bytes:655 tcp.segments frames:2 bytes:655
The PHS output show that every ERF frame contains an Ethernet frame (eth). We can therefore specify the output filetype to be libpcap and encapsulation type to be and Ethernet like this:
editcap -F libpcap -T ether erf-ethernet-example.erf erf-ethernet-example.pcap
Let's check the PHS for the pcap file we've now generated:
tshark -q -z io,phs -r erf-ethernet-example.pcap
Protocol Hierarchy Statistics
eth frames:19 bytes:7269 ip frames:19 bytes:7269 tcp frames:19 bytes:7269 http frames:4 bytes:2077 image-gif frames:2 bytes:655 tcp.segments frames:2 bytes:655
Success! All ERF headers are now gone and the output pcap file contains plain old Ethernet frames. You can now open the pcap in NetworkMiner or whichever pcap parsing tool you wish.
NetworkMiner with erf-ethernet-example.pcap loaded
Another way to convert an ERF file to the PCAP or PcapNG format is to carve packets from the ERF file with CapLoader.
Posted by Erik Hjelmvik on Thursday, 22 November 2012 13:11:00 (UTC/GMT)
There was recently a question on the Wireshark users mailing list about “how to get the query name from a dns request packet with tshark”. This is a problem that many network analysts run into, so I decided to write a blog post instead of just replying to the mailing list.
Note: the pcap file used in this blog post is from the DFRWS 2009 Challenge.
Who queried for a particular domain?
Tshark can easily be used in order to determine who queried for a particular domain, such as google.com, by using the following command:
tshark -r nssal-capture-1.pcap -T fields -e ip.src -e dns.qry.name -R "dns.flags.response eq 0 and dns.qry.name contains google.com"
List all queries
A list of ALL queries can be built with the same command, but without filtering on a particular domain:
tshark -r nssal-capture-1.pcap -T fields -e ip.src -e dns.qry.name -R "dns.flags.response eq 0"
DNS lists in NetworkMiner
There is a DNS tab in NetworkMiner, which displays a nice list of all DNS queries and responses in a pcap file. Loading the same nssal-capture-1.pcap into NetworkMiner generates the following list:
DNS tab with nssal-capture-1.pcap loaded
NetworkMiner Professional also has the ability to export this data to a CSV file. The command line tool NetworkMinerCLI can also generate such a CSV file without a GUI, which is perfect if you wanna integrate it in a customized script.
Posted by Erik Hjelmvik on Sunday, 17 June 2012 17:45:00 (UTC/GMT)
This blog post is the second part of a two-part tutorial that shows how to sniff network traffic. The first part covered how to intercept the traffic, i.e. how to get the packets to arrive to your network card. This second part covers how to best capture the network traffic to disk once you've managed to have them sent to your network card.
The first thing I recommend you to do before you start sniffing is to ensure that your sniffer computer stays quiet, so that you don't pollute the sniffed traffic with packets from your sniffer computer. How to do this varies between Windows, Linux and FreeBSD, so I will briefly cover how to silence all three OS's.
Windows is normally quite chatty on the network, but there is a simple trick that makes it shut up without disconnecting it from the network. Simply bring up the properties window for your network adapter and un-check all clients, services and protocols from the list of used items.
I don't know any good way to disable the TCP/IP stack in Linux similar to what we just did to Windows. But by disabling ARP and not giving the interface any IP address the risk of leaking packets can be reduced significantly. To achieve this you'll also need to set the interface to promiscuous mode and disable IPv6. Here's an example of how this can be done in Debian/Ubuntu:
# ifconfig eth0 -arp upHowever, in general it is better to configure the sniffer interface by modifying /etc/network/interfaces to look as follows:
# ifconfig eth0 promisc
# ip addr del 192.168.3.4/24 dev eth0
# ifconfig eth0 del fe80::1234:ddff:fe3f:1337/64
iface eth0 inet manual
up ifconfig $IFACE -arp up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down
For more details on shuting up your Linux machine I suggest you take a look at the “Undetectable Sniffing On Ethernet” blog post on the Ask Apache blog. I also recommend reading the Ubuntu instructions at the NetworkConfiguration wiki page for Security Onion.
One of the many great things about FreeBSD is that it is really easy to configure a network interface to be silent. All you need to do is to issue this single command:
# ifconfig em0 monitor up
Read Richard Bejtlich's blog post “Ifconfig Monitor Option” for more info about this command.
Sniff with dumpcap
The most common software used for sniffing traffic is undoubtedly Wireshark, other common tools are tshark and tcpdump. All these three tools parse the packets as they come in and show data about the sniffed packets in the GUI or on the command line. This isn't desirable when doing network forensics since these application could use up too much resources on parsing the incoming packets, which during traffic peaks can lead to packets being dropped. There might also be bugs in the packet parsing code that can cause the application to crash completely, there have been plenty of such bugs in Wireshark for example.
A much more reliable way to sniff traffic is therefore to use an application that is designed to reliably dump traffic to disk, without trying to parse or analyze the contents of the sniffed frames. One such application is dumpcap, which is a command line tool developed by the Wireshark crew. In fact, as of version 1.4.0 of Wireshark both tshark and Wireshark run dumpcap in the background rather than doing the sniffing themselves. Another great thing about dumpcap is that it is available on Windows as well as on Linux and FreeBSD. My recommendation is to run it under FreeBSD since I consider it to be more reliable than the other operating systems. In order to get dumcap you need to download either Wirehsark or tshark since dumpcap doesn't have an installation package of its own.
Dumpcap supports capture filtering with BPF, but I recommend that you sniff ALL packets if you are authorized to do so and have enough disk space. It is much better to do the filtering later on in the analysis phase. The feature I like best in dumpcap is the ring buffer functionality, where it will continue writing incoming frames to a new pcap file once the old one has reached a specific size limit. In order to have dumpcap write sniffed frames to disk and roll over to a new file when it reaches 100.000 kB you'd run this command:
# dumpcap -i em0 -w wiretap.pcap -b filesize:100000
When you've dumped the traffic to disk is when the fun starts, i.e. analyzing the pcap files. You can read more analyzing pcap files in my blog posts about the TCP/IP Weapons School Sample Lab, DFRWS 2009 Challenge, Webmail Information Leakage and Command-line forensics.
Posted by Erik Hjelmvik on Friday, 11 March 2011 16:56:00 (UTC/GMT)