NETRESEC Network Security Blog - Tag : PcapNG
We were contacted by GreatFire.org earlier today regarding a new Chinese man-in-the-middle (MITM) attack. This time the perpetrators decrypted traffic between Chinese users and Microsoft's IMAP mail server for outlook.com. As evidence GreatFire.org provided us with a packet capture file, which we have analyzed.
Our conclusion is that this was a real attack on Microsoft's email service. Additionally, the attack is very similar to previous nationwide Chinese attacks on SSL encrypted traffic, such as the attack on Google a few months ago. Details such as email address, email password, email contents, email attachments and contacts may have been compromised in this attack. We do not know the scale of the attack, it could be anything from a fairly targeted attack to a nation wide attack in China. What we do know is that there are several users who have been subjected to the MITM attack and posted screenshots online.
|Attacked IP Address:||18.104.22.168 (imap-mail.outlook.com)|
|Attacked Protocol:||SSL encryption of IMAPS (TCP 993)|
|Date of Attack:||2015-01-18|
In our technical analysis we first extracted the x509 certificates from the SSL traffic by loading the capture file into NetworkMinerCLI. We then parsed the extracted certificates with OpenSSL.
$ mono /opt/NetworkMinerProfessional_1-6-1/NetworkMinerCLI.exe -r Outlook_MITM_2015-01-18.pcapng
Closing file handles...
84 frames parsed in 0.888754 seconds.
$ ls AssembledFiles/22.214.171.124/TLS_Cert\ -\ TCP\ 993/*.cer
AssembledFiles/126.96.36.199/TLS_Cert - TCP 993/hotmail.com.cer
AssembledFiles/188.8.131.52/TLS_Cert - TCP 993/hotmail.com.cer
AssembledFiles/184.108.40.206/TLS_Cert - TCP 993/hotmail.com.cer
$ openssl x509 -inform DER -in AssembledFiles/220.127.116.11/TLS_Cert\ -\ TCP\ 993/hotmail.com.cer -noout -issuer -subject -startdate -fingerprint
notBefore=Jan 15 16:00:00 2015 GMT
When looking at the timestamps in the capture file we noticed that the SSL server's reply to the 'Client Hello' was very slow; response times varied between 14 and 20 seconds. Under normal circumstances the 'Server Hello' arrives within 0.3 seconds after the 'Client Hello' has been sent.
$ tshark -nr ./Outlook_MITM_2015-01-18.pcapng -Y 'ssl.handshake.type lt 3'
8 9.023876000 10.0.2.15 -> 18.104.22.168 SSL 265 Client Hello
17 26.885504000 22.214.171.124 -> 10.0.2.15 TLSv1 576 Server Hello, Certificate, Server Hello Done
45 101.747755000 10.0.2.15 -> 126.96.36.199 SSL 265 Client Hello
49 116.258483000 188.8.131.52 -> 10.0.2.15 TLSv1 576 Server Hello, Certificate, Server Hello Done
63 116.338420000 10.0.2.15 -> 184.108.40.206 SSL 265 Client Hello
65 136.119127000 220.127.116.11 -> 10.0.2.15 TLSv1 576 Server Hello, Certificate, Server Hello Done
This is slow SSL response is consistent with previous SSL MITM attacks conducted with support of the Great Firewall of China (GFW).
For more details on this attack, please see the Reuters story "After Gmail blocked in China, Microsoft's Outlook hacked" and GreatFire's own blog post "Outlook grim - Chinese authorities attack Microsoft".
Posted by Erik Hjelmvik on Monday, 19 January 2015 22:55:00 (UTC/GMT)
GreatFire.org, who monitor the Great Firewall of China (GFW), also published a
blog post on their website earlier today saying:
This is clearly a malicious attack on Apple in an effort to gain access to usernames and passwords and consequently all data stored on iCloud such as iMessages, photos, contacts, etc.
Fake SSL Certificate
In their blog post GreatFire also linked a packet capture file, which we have analyzed in order to verify the MITM attack. We loaded the PcapNG file into NetworkMiner Professional and extracted the X.509 SSL certificate.
The extracted certificate can be downloaded from here. Also, here are a few details from this X.509 certificate:
$ openssl x509 -inform DER -in www.icloud.com.cer -noout -issuer -subject -startdate -enddate -fingerprint
notBefore=Oct 4 10:35:47 2014 GMT
notAfter=Oct 4 10:35:47 2015 GMT
As reported elsewhere, the certificate was self signed, which means that browsers and most iPhone apps will either inform the user about the connection being unsafe or simply close the connection (see update at the bottom of this blog post regarding the missing certificate verification in Apple iOS). This use of self signed certificates is consistent with previous SSL MITM attacks performed in China against GitHub, Google, Yahoo and live.com.
Location of the MITM Attack
By looking at host the information provided by NetworkMiner for the fake iCloud SSL server we can see that it is just six router hops away from the client (having an IP TTL value of 58). This indicates that the MITM attack is being performed within China, since we'd expect to see at least three more router hops if the packets were coming from outside China.
The same PCAP file also contains packets from the same IP address on TCP port 80, which have traveled 11 hops (IP TTL 53). We therefore assume that only traffic to TCP port 443 is being MITM'ed.
This TTL analysis also matches various TCP traceroutes we've seen to the MITM'ed iCloud SSL service on 18.104.22.168:443.
My traceroute [v0.85]mtr TCP 443 traceroute to 22.214.171.124 (source: http://pastebin.com/8Y6ZwfzG)
siyanmao-k29 (0.0.0.0) Sat Oct 18 19:26:07 2014
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.1 0.0% 17 0.6 0.7 0.6 0.8 0.0
2. ------------- 0.0% 16 2.8 2.6 1.7 3.3 0.3
3. ------------- 0.0% 16 2.0 2.2 1.4 4.0 0.4
5. 126.96.36.199 0.0% 16 6.4 7.7 4.3 27.0 5.2
6. 188.8.131.52 25.0% 16 168.5 171.4 166.8 201.3 9.4
The mtr TCP traceroute above indicates that MITM attacks are performed in AS4134 (China Telecom).
bearice@Bearice-Mac-Air-Haswell ~tcptraceroute to 184.108.40.206 443 (source: bearice on GitHub)
%tcptraceroute 220.127.116.11 443
Selected device en0, address 192.168.100.16, port 52406 for outgoing packets
Tracing the path to 18.104.22.168 on TCP port 443 (https), 30 hops max
1 192.168.100.254 1.737 ms 0.793 ms 0.798 ms
2 22.214.171.124 2.893 ms 2.967 ms 2.422 ms
3 126.96.36.199 2.913 ms 2.893 ms 3.968 ms
4 188.8.131.52 4.824 ms 2.658 ms 3.902 ms
5 184.108.40.206 3.626 ms 6.532 ms 3.794 ms
6 220.127.116.11 27.539 ms 26.821 ms 27.661 ms
7 a23-59-94-46.deploy.static.akamaitechnologies.com (18.104.22.168) [open] 30.064 ms 29.899 ms 30.126 ms
The tcptraceroute above indicates that MITM attacks are also performed in AS4837 (China Unicom).
The Tcproute screenshot above shows that also CHINANET backbone network (China Telecom) seems to be used to carry out the MITM attacks.
Judging from these TCP traceroutes the MITM attacks seem to be taking place at several different locations rather centrally in the Chinese Internet infrastructure. To be more specific, it appears as if the MITM attacks are being performed on backbone networks belonging to China Telecom (CHINANET) as well as China Unicom.
UPDATE (October 22)
A vulnerability notice (CVE-2014-4449) has now been published, where Apple confirm that fake SSL certificates (like the Chinese fake one) were not verified by Apple iOS before 8.1. Apple released the first details about this vulnerability just a few hours after this blog post was published. Here's the text from the CVE description:
iCloud Data Access in Apple iOS before 8.1 does not verify X.509 certificates from TLS servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.This means that the Chinese MITM of iCloud could potentially have revealed a significant number of iCloud credentials as well as private data (images, videos, documents etc) to the attackers. Or, as @Exploit_This tweeted: "So china wants our nudes?"
Posted by Erik Hjelmvik on Monday, 20 October 2014 13:35:00 (UTC/GMT)
GreatFire.org sent out a tweet yesterday saying that “Yahoo appears to under Man-in-the-middle attack in China. 3rd case of country-wide MITM, after Google, Github”.
Mashable later ran a story called “China Appears to Attack Yahoo in Latest Censorship of Hong Kong Protests”, where Lorenzo Franceschi-Bicchierai write:
In what's almost unprecedented, China appears to be targeting Yahoo with what's called a "man-in-the-middle attack." With such an attack, connections to Yahoo.com, which are normally encrypted, would be vulnerable to snooping, and Chinese censors could also block search terms or specific Yahoo links with the goal of preventing Chinese netizens from accessing information about the protests in Hong Kong.
In this blog post we verify that there is an ongoing Man-in-the-Middle (MITM) attack by analyzing two different packet capture files.
|Capture Location||Capture Date||Filename||MD5|
Both PCAP files were created with Wireshark/dumpcap using a capture filter of “host 22.214.171.124”, which is the IP address that was reported to be MITM'ed by the Great Firewall of China (GFW). This IP address is located in Hong Kong and is used by Yahoo to host www.yahoo.com, hk.yahoo.com etc. for users in this part of Asia.
Time-To-Live (TTL) Analysis
We estimate the distance between the end users and the Yahoo server in Hong Kong to be at least 10 router hops. However, the IP packets coming back to the users have IP TTL values of 58 (Wuxi) and 57 (Zhengzhou). This implies that the MITM is performed somewhere in China, just 6 or 7 router hops away from the users. This is consistent with what we've observed in previous MITM attacks performed by China against GitHub and Google.
IMAGE: Hosts tab in CapLoader showing TTL 57 for hk.yahoo.com:443
X.509 Certificate Analysis
$ openssl x509 -inform DER -in yahoo.com.cer -noout -issuer -subject -startdate -enddate -fingerprint
notBefore=Sep 23 11:30:17 2014 GMT
notAfter=Sep 23 11:30:17 2015 GMT
The certificate is a self signed certificate for “yahoo.com”. The fact that the MITM uses a self signed certificate makes the attack easily detectable even for the non-technical user, since the web browser will typically display a warning about the site not being trusted.
Some may think it's odd that China can't forge a properly signed certificate for their SSL MITM attack. However, they've used very similar self signed certificates also in their previous MITM attacks against GitHub and Google. The purpose of GFW (a.k.a. “Golden Shield”) is to censor the Internet, so the primary goal with this MITM attack isn't to covertly spy on Chinese Yahoo searches. Regardless if the end users notice the MITM or not, a self signed X.509 cert is enough in order to see what they are searching for and “kill” their connection to Yahoo when queries like “Umbrella Revolution” and “Tiananmen Square Protests” are observed.
Posted by Erik Hjelmvik on Wednesday, 01 October 2014 21:55:00 (UTC/GMT)
The Chinese are running a MITM attack on SSL encrypted traffic between Chinese universities and Google. We've performed technical analysis of the attack, on request from GreatFire.org, and can confirm that it is a real SSL MITM against www.google.com and that it is being performed from within China.
We were contacted by GreatFire.org yesterday (September 3) with a request to analyze two packet captures from suspected MITM-attacks before they finalized their blog post. The conclusions from our analysis is now published as part of GreatFire.org's great blog post titled “Authorities launch man-in-the-middle attack on Google”.
In their blog post GreatFire.org write:
From August 28, 2014 reports appeared on Weibo and Google Plus that users in China trying to access google.com and google.com.hk via CERNET, the country’s education network, were receiving warning messages about invalid SSL certificates. The evidence, which we include later in this post, indicates that this was caused by a man-in-the-middle attack.
While the authorities have been blocking access to most things Google since June 4th, they have kept their hands off of CERNET, China’s nationwide education and research network. However, in the lead up to the new school year, the Chinese authorities launched a man-in-the-middle (MITM) attack against Google.
Our network forensic analysis was performed by investigating the following to packet capture files:
|Capture Location||Client Netname||Capture Date||Filename||MD5|
|Peking University||PKU6-CERNET2||Aug 30, 2014||google.com.pcap||aba4b35cb85ed218 7a8a7656cd670a93|
|Chongqing University||CQU6-CERNET2||Sep 1, 2014||google_fake.pcapng||3bf943ea453f9afa 5c06b9c126d79557|
Client and Server IP adresses
The analyzed capture files contain pure IPv6 traffic (CERNET is a IPv6 network) which made the analysis a bit different then usual. We do not disclose the client IP addresses for privacy reasons, but they both seem legit; one from Peking University (netname PKU6-CERNET2) and the other from Chongqing University (CQU6-CERNET2). Both IP addresses belong to AS23910, named "China Next Generation Internet CERNET2".
The IP addresses received for www.google.com were in both cases also legit, so the MITM wasn't carried out through DNS spoofing. The Peking University client connected to 2607:f8b0:4007:804::1013 (GOOGLE-IPV6 in United States) and the connection from Chongqing University went to 2404:6800:4005:805::1010 (GOOGLE_IPV6_AP-20080930 in Australia).
Time-To-Live (TTL) Analysis
The Time-To-Live (TTL) values received in the IP packets from www.google.com were in both cases 248 or 249 (note: TTL is actually called ”Hop Limit” in IPv6 nomenclature, but we prefer to use the well established term ”TTL” anyway). The highest possible TTL value is 255, this means that the received packets haven't made more than 6 or 7 router hops before ending up at the client. However, the expected number of router hops between a server on GOOGLE-IPV6 and the client at Peking University is around 14. The low number of router hops is is a clear indication of an IP MITM taking place.
Image: CapLoader with both capture files loaded, showing TTL values
Here is an IPv6 traceroute from AS25795 in Los Angeles towards the IP address at Peking University (generated with ARP Networks' 4or6.com tool):
#traceroute -6 2001:da8:201:[REDACTED]
1 2607:f2f8:1600::1 (2607:f2f8:1600::1) 1.636 ms 1.573 ms 1.557 ms
2 2001:504:13::1a (2001:504:13::1a) 40.381 ms 40.481 ms 40.565 ms
3 * * *
4 2001:252:0:302::1 (2001:252:0:302::1) 148.409 ms 148.501 ms 148.595 ms
5 * * *
6 2001:252:0:1::1 (2001:252:0:1::1) 148.273 ms 147.620 ms 147.596 ms
7 pku-bj-v6.cernet2.net (2001:da8:1:1b::2) 147.574 ms 147.619 ms 147.420 ms
8 2001:da8:1:50d::2 (2001:da8:1:50d::2) 148.582 ms 148.670 ms 148.979 ms
9 cernet2.net (2001:da8:ac:ffff::2) 147.963 ms 147.956 ms 147.988 ms
10 2001:da8:201:[REDACTED] 147.964 ms 148.035 ms 147.895 ms
11 2001:da8:201:[REDACTED] 147.832 ms 147.881 ms 147.836 ms
12 2001:da8:201:[REDACTED] 147.809 ms 147.707 ms 147.899 ms
As can be seen in the traceroute above, seven hops before the client we find the 2001:252::/32 network, which is called “CNGI International Gateway Network (CNGIIGN)”. This network is actually part of CERNET, but on AS23911, which is the network that connects CERNET with its external peers. A reasonable assumption is therefore that the MITM is carried out on the 2001:252::/32 network, or where AS23910 (2001:da8:1::2) connects to AS23911 (2001:252:0:1::1). This means that the MITM attack is being conducted from within China.
Response Time Analysis
The round-trip time between the client and server can be estimated by measuring the time from when the client sends it initial TCP SYN packet to when it receives a TCP SYN+ACK from the server. The expected round-trip time for connecting from CERNET to a Google server overseas would be around 150ms or more. However, in the captures we've analyzed the TCP SYN+ACK package was received in just 8ms (Peking) and 52ms (Chongqing) respectively. Again, this is a clear indication of an IP MITM taking place, since Google cannot possibly send a response from the US to CERNET within 8ms regardless of how fast they are. The fast response times also indicate that the machine performing the MITM is located fairly close to the network at Peking University.
Even though the machine performing the MITM was very quick at performing the TCP tree-way handshake we noticed that the application layer communication was terribly slow. The specification for the TLS handshake (RFC 2246) defines that a ClientHello message should be responded to with a ServerHello. Google typically send their ServerHello response almost instantly, i.e. the response is received after one round-trip time (150ms in this case). However, in the analyzed captures we noticed ServerHello response times of around 500ms.
X.509 Certificate Analysis
The X.509 certificates were extracted from the two PCAP files to .cer files using NetworkMiner. We noticed that both users received identical certificates, which were both self signed for “google.com”. The fact that the MITM used a self signed certificate makes the attack easily detectable even for the non-technical user, since the web browser will typically display a warning about the site not being trusted. Additionally the X.509 certificate was created for ”google.com” rather than ”*.google.com”. This is an obvious miss from the MITM'ers side since they were attempting to MITM traffic to ”www.google.com” but not to ”google.com”.
Image: NetworkMiner showing list of X.509 certificates extracted from the two PCAP files
Certificate SHA1 fingerprint: f6beadb9bc02e0a152d71c318739cdecfc1c085d
Certificate MD5 fingerprint: 66:D5:D5:6A:E9:28:51:7C:03:53:C5:E1:33:14:A8:3B
A copy of the fake certificate is available on Google drive thanks to GreatFire.org.
All evidence indicates that a MITM attack is being conducted against traffic between China’s nationwide education and research network CERNET and www.google.com. It looks as if the MITM is carried out on a network belonging to AS23911, which is the outer part of CERNET that peers with all external networks. This network is located in China, so we can conclude that the MITM was being done within the country.
It's difficult to say exactly how the MITM attack was carried out, but we can dismiss DNS spoofing as the used method. The evidence we've observed instead indicate that the MITM attack is performed either by performing IP hijacking or by simply reconfiguring a router to forward the HTTPS traffic to a transparent SSL proxy. An alternative to changing the router config would also be to add an in-line device that redirects the desired traffic to the SSL proxy. However, regardless of how they did it the attacker would be able to decrypt and inspect the traffic going to Google.
We can also conclude that the method used to perform the MITM attack was similar to the Chinese MITM on GitHub, but not identical.
Posted by Erik Hjelmvik on Thursday, 04 September 2014 23:55:00 (UTC/GMT)
A new feature in the recently released CapLoader 1.2 is the ability to carve network packets from any file and save them in the PCAP-NG format. This fusion between memory forensics and network forensics makes it possible to extract sent and received IP frames, with complete payload, from RAM dumps as well as from raw disk images.
CapLoader will basically carve any TCP or UDP packet that is preceded by an IP frame (both IPv4 and IPv6 are supported), and believe me; there are quite a few such packets in a normal memory image!
We've made the packet carver feature available in the free version of CapLoader, so feel free to give it a try!
The packet carving feature makes it possible do much better analysis of network traffic in memory dumps compared to Volatility's connscan2. With Volatility you basically get the IP addresses and port numbers that communicated, but with CapLoader's packet carver you also get the contents of the communication!
EXAMPLE: Honeynet Banking Troubles Image
I loaded the publicly available “Banking Troubles” memory image from the Honeynet Project into CapLoader to exemplify the packet carver's usefulness in a digital forensics / incident response (DFIR) scenario.
CapLoader 1.2 Carving Packets from HoneyNet Memory Image
22 TCP/UDP Flows were carved from the memory image by CapLoader
Let's look at the network traffic information that was extracted in the Honeynet Project's own solution for the Banking Troubles Challenge:
python volatility connscan2 -f images/hn_forensics.vmem"
Local Address Remote Address Pid
------------------------- ------------------------- ------
192.168.0.176:1176 126.96.36.199:80 888
192.168.0.176:1189 192.168.0.1:9393 1244
192.168.0.176:2869 192.168.0.1:30379 1244
192.168.0.176:2869 192.168.0.1:30380 4
0.0.0.0:0 188.8.131.52:0 0
127.0.0.1:1168 127.0.0.1:1169 888
192.168.0.176:1172 184.108.40.206:80 888
127.0.0.1:1169 127.0.0.1:1168 888
192.168.0.176:1171 220.127.116.11:80 888
192.168.0.176:1178 18.104.22.168:80 1752
192.168.0.176:1184 22.214.171.124:80 880
192.168.0.176:1185 126.96.36.199:80 880
"This connection [marked in bold above] was opened by AcroRd32.exe (PID 1752) and this represents an additional clue that an Adobe Reader exploit was used in order to download and execute a malware sample."
The solution doesn't provide any evidence regarding what Acrobat Reader actually used the TCP connection for. Additionally, none of the three finalists managed to prove what was sent over this connection.
To view the payload of this TCP connection in CapLoader, I simply right-clicked the corresponding row and selected “Flow Transcript”.
Transcript of TCP flow contents (much like Wireshark's Follow-TCP-Stream)
We can see that the following was sent from 192.168.0.176 to 188.8.131.52:
GET /load.php?a=a&st=Internet%20Explorer%206.0&e=2 HTTP/1.1
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Notice that the HTTP GET request took place at the end of the TCP session. Odd? Well, CapLoader doesn't know the timestamp of carved packets, so they are simply ordered as they were discovered in the dump file. The timestamp generated for each carved packet represents where in the image/dump the packet was found. Or more precise: the number of microseconds since EPOCH (1970-01-01 00:00:00) is the offset (in bytes) from where the packet was carved.
Hence, we know that the HTTP GET request can be found between offset 37068800 and 37507072 in the image (a 428 kB region). To be more exact we can open the generated PcapNG file with Wireshark or Tshark to get the timestamp and length of the actual HTTP GET request packet.
tshark.exe -r Bob.vmem.pcapng" -R http.request -T fields -e frame.time_epoch -e frame.len -e http.request.uri
31.900664000 175 *
37.457920000 175 *
37.462016000 286 /load.php?a=a&st=Internet%20Explorer%206.0&e=2
37.509120000 175 *
37.519360000 245 /~produkt/983745213424/34650798253
37.552128000 266 /root.sxml
37.570560000 265 /l3fw.xml
37.591040000 274 /WANCommonIFC1.xml
37.607424000 271 /WANIPConn1.xml
Now, lets verify that the raw packet data is actually 37462016 bytes into the memory dump.
xxd -s 37462016 -l 286 Bob.vmemYep, that's our HTTP GET packet preceded by an Ethernet, IP and TCP header.
23ba000: 0021 9101 b248 000c 2920 d71e 0800 4500 .!...H..) ....E.
23ba010: 0110 3113 4000 8006 8e1a c0a8 00b0 d496 ..1.@...........
23ba020: a4cb 049a 0050 7799 0550 f33b 7886 5018 .....Pw..P.;x.P.
23ba030: faf0 227e 0000 4745 5420 2f6c 6f61 642e .."~..GET /load.
23ba040: 7068 703f 613d 6126 7374 3d49 6e74 6572 php?a=a&st=Inter
23ba050: 6e65 7425 3230 4578 706c 6f72 6572 2532 net%20Explorer%2
23ba060: 3036 2e30 2665 3d32 2048 5454 502f 312e 06.0&e=2 HTTP/1.
23ba070: 310d 0a41 6363 6570 743a 202a 2f2a 0d0a 1..Accept: */*..
23ba080: 4163 6365 7074 2d45 6e63 6f64 696e 673a Accept-Encoding:
23ba090: 2067 7a69 702c 2064 6566 6c61 7465 0d0a gzip, deflate..
23ba0a0: 5573 6572 2d41 6765 6e74 3a20 4d6f 7a69 User-Agent: Mozi
23ba0b0: 6c6c 612f 342e 3020 2863 6f6d 7061 7469 lla/4.0 (compati
23ba0c0: 626c 653b 204d 5349 4520 362e 303b 2057 ble; MSIE 6.0; W
23ba0d0: 696e 646f 7773 204e 5420 352e 313b 2053 indows NT 5.1; S
23ba0e0: 5631 290d 0a48 6f73 743a 2073 6561 7263 V1)..Host: searc
23ba0f0: 682d 6e65 7477 6f72 6b2d 706c 7573 2e63 h-network-plus.c
23ba100: 6f6d 0d0a 436f 6e6e 6563 7469 6f6e 3a20 om..Connection:
23ba110: 4b65 6570 2d41 6c69 7665 0d0a 0d0a Keep-Alive....
Give it a Try!
Wanna verify the packet carving functionality? Well, that's easy! Just follow these three steps:
Download a sample memory image (thanks for the great resource Volatility Team!)
Download the free RAM dumper DumpIt and dump your own computer's memory.
Locate an existing file that already contains parts of your RAM, such as pagefile.sys or hiberfil.sys
- Download the free version of CapLoader and open the memory dump.
Select destination for the generated PcapNG file with carved packets and hit the “Carve” button!
Carving Packets from Proprietary and odd Capture Formats
CapLoader can parse PCAP and PcapNG files, which are the two most widely used packet capture formats. However, the packet carving features makes it possible to extract packets from pretty much any capture format, including proprietary ones. The drawback is that timestamp information will be lost.
We have successfully verified that CapLaoder can carve packets from the following network packet capture / network trace file formats:
- .ETL files created with netsh or logman. These Event Trace Log files can be created without having WinPcap installed.
- .CAP files created with Microsoft Network Monitor
- .ENC files (NA Sniffer) from IBM ISS products like the Proventia IPS (as well as Robert Graham's old BlackICE)
- .ERF files from Endace probes
Posted by Erik Hjelmvik on Monday, 17 March 2014 10:05:00 (UTC/GMT)
NetworkMiner 1.5 was released on August 7th, but we haven't yet provided any details regarding the new functionality that has been added.
NetworkMiner (free edition)
New features in the free and open source version of NetworkMiner:
- Parser for PPPoE (RFC 2615)
- Keywords can be loaded from text file (useful in investigations where you have lots of strings to search for)
- Support for LLMNR DNS (RFC 4795) queries over UDP 5355
The professional version of NetworkMiner additionally contains the following new features:
- NetworkMinerCLI generates a Keywords CSV file when one or several keywords are detected
- NetworkMinerCLI can read a custom keyword list and cleartext dictionary from file using command line arguments
- Parsing of PcapNG (aka pcap-ng) files
- Extraction of metadata from PcapNG files (including stored name resolution blocks)
- Alexa top 1M check for DNS responses
DNS tab in NetworkMiner Professional with eee.pcapng from CloudShark loaded.
Metadata window in NetworkMiner Professional
In order to bring up the metadata window, simply right-click a PcapNG file in NetworkMiner's case panel and select "Show Metadata". The extracted metadata will normally contain info about the machine used to create the capture file, such as the OS and what sniffer that was used. Also, please note that some PcapNG files additionally contain name resolution blocks with cached DNS entries even if the corresponding traffic has been filtered from the capture file.
We hope you will find these new features useful!
Posted by Erik Hjelmvik on Sunday, 15 September 2013 21:03:00 (UTC/GMT)
In our blog post about the Chinese MITM of GitHub we revealed the identity of the anonymous capture file uploader by analyzing metadata available in the PCAP-NG file format. In this blog post we explain what type of meta-data that can be found in PcapNG files, and how to extract it.
The old libpcap format (a.k.a. PCAP) is designed to only store captured network frames. The new PCAP-NG format, however, additionally includes the ability to store meta-data in the capture files. In fact only about 20% of the PcapNG file specification concerns storage of captured frames, the remaining 80% is focused on various types of metadata.
So what types of metadata can a PcapNG file contain? Well, the spec allows features such as MAC address of capturing interface as well as interface speed (in bps) to be stored. But the most commonly used metadata types are:
- Operating system of the sniffer machine
- Software used to create the capture file (application name and version)
- Name of interface from where packets are captured
- Description of interface from where packets are captured
- Capture filter used when sniffing packets
- Cached name resolution entries (mappings from IPv4 or IPv6 addresses to host names)
- User comments on individual frames (a.k.a. “annotations”),
Some of these metadata types can be displayed in Wireshark by clicking “Statistics > Summary”. Here's the information provided when displaying a Summary in Wireshark on MachineB.pcapng from CloudShark:
However, what is not shown in this summary view is that MachineB.pcapng additionally contains cached name resolution information for two machines. The easiest way to extract this extra metadata is to upload the capture file to pcapng.com, which will display a list of the metadata that was found in the uploaded file as well as convert the capture file to the old libpcap format (without metadata).
Here's the list of metadata information extracted from MachineB.pcapng by pcapng.com:
|shb_os||Windows Server 2003 Service Pack 2, build 3790|
|shb_userappl||Dumpcap 1.8.4 (SVN Rev 46250 from /trunk-1.8)|
|if_os||Windows Server 2003 Service Pack 2, build 3790|
|nres_ip6record||fe80::9a03:d8ff:fedb:904b = iPhone-von-b1.local|
|nres_ip4record||192.168.13.124 = iPhone-von-b1.local|
|nres_ip6record||fe80::8685:6ff:fe23:c95d = iPhone-von-Gurkan.local|
|nres_ip4record||192.168.13.188 = iPhone-von-Gurkan.local|
The metadata apparently contains name resolution info for two iPhones, and we've got their internal IPv4 as well as IPv6 addresses. This would normally imply that the capture file contains frames to or from these IP addresses. However, as shown in the following screenshot from CloudShark, the capture file contains 21 frames but not a single one of them is from or to “iPhone-von-Gurkan” or “iPhone-von-b1”:
Thus, it seems as if the Wireshark pcap-ng host name disclosure bug wasn't fixed in Wireshark 1.8.4 after all.
Megecap Leaks File Paths
It is, however, not only Wireshark that might leak private information in PcapNG metadata. The command line tool Mergecap additionally stores the complete path and filename of the individual capture files that have been combined with the tool.
|opt_comment||File created by merging: File1: /home/erik/captures/customer_X/capture_tap0_p5p1.cap File2: /home/erik/captures/customer_X/capture_tap1_p1p1.cap|
Metadata available in PcapNG file created with mergecap
This is most likely not a bug, but rather information leakage by design, so don't expect this to be mitigated in a future Mergecap release.
As we pointed out in our blog post “HowTo handle PcapNG files”,
the best way to avoid information leakage via PcapNG files is to stay with the good ol' PCAP format
by using the “-P” switch like this:
dumpcap -P -i eth0 -w dump.pcap
Another alternative is to convert your PcapNG files to PCAP format before sharing them.
Posted by Erik Hjelmvik on Thursday, 14 February 2013 19:25:00 (UTC/GMT)
On January 26 several users in China reported SSL problems while connecting to the software development site GitHub.com. The reports indicated that the Great Firewall of China (GFW) was used to perform a Man-in-the-Middle (MITM) attack against users in China who were visiting GitHub. This attack was most likely conducted in order to track which users that were reading or contributing to the list of persons behind the GFW, which is hosted on GitHub.
There is a good writetup on the attack by GreatFire.org, which says:
”At around 8pm, on January 26, reports appeared on Weibo and Twitter that users in China trying to access GitHub.com were getting warning messages about invalid SSL certificates. The evidence, listed further down in this post, indicates that this was caused by a man-in-the-middle attack.
It was very crude, in that the fake certificate was signed by an unknown authority and bound to be detected quickly. The attack stopped after about an hour.”
Screenshot of SSL error at github.com by @bitinn
A self-signed X.509 certificate file, named github.com.crt, was also used as evidence. It is, however, easy to extract X.509 certificates from PCAP files, so the github.com.crt file might come from the github.pcapng capture file at CloudShark.
We have no reason not to trust the reports coming from users in China, who have been victims of this MITM attack. However, in the spirit of “Trust, but verify” I decided to perform some network forensic analysis on the capture file. More specifically I set out to answer the following questions:
|Is the user a victim of a real attack rather than having staged and recorded a MITM attack set up by himself?|
|Is the user located in China?|
|What can we say about the technology being used for the MITM?|
My first step was to load the PcapNG file from CloudShark in CapLoader:
File > Open URL > Enter:
The “Hosts” tab shows that the capture file only contained traffic between two network hosts; the user's machine and the GitHub server. I also noticed that the GitHub server (184.108.40.206) had an IP TTL of 58 and TCP Window Size of 5840. This values are a typical indication of the host being a Linux machine (see our passive OS fingerprinting blog post for more details). However, when connecting to GitHub.com from here (Sweden) the same server has an IP TTL of 128 and Window Size 64240. This looks more like a Windows machine, and is significantly different from what the Chinese user got.
It can also be noted that a TTL of 58, which the Chinese user got, would mean that the SSL connection was terminated just six router hops away from the user. However, a traceroute from ihep.ac.cn (in Beijing, China) to github.com requires a whopping 16 hops. Additionally, this traceroute hasn't even left Beijing after six hops.
traceroute to 220.127.116.11 (18.104.22.168), 30 hops max, 38 byte packets
1 gw (22.214.171.124) 0.231 ms 0.175 ms 0.178 ms
2 126.96.36.199 (188.8.131.52) 0.719 ms 0.626 ms 0.652 ms
3 192.168.1.25 (192.168.1.25) 0.509 ms 0.485 ms 0.459 ms
4 8.131 (184.108.40.206) 0.516 ms 0.488 ms 0.496 ms
5 8.198 (220.127.116.11) 11.981 ms 0.913 ms 0.829 ms
6 8.192 (18.104.22.168) 40.477 ms 40.614 ms 40.524 ms
7 Gi6-0.gw2.hkg3.asianetcom.net (22.214.171.124) 53.062 ms 40.610 ms 40.617 ms
8 te0-3-0-0.wr1.hkg0.asianetcom.net (126.96.36.199) 42.459 ms 43.133 ms 43.765 ms
9 te0-1-0-0.wr1.osa0.asianetcom.net (188.8.131.52) 76.603 ms 76.586 ms 76.648 ms
10 gi6-0-0.gw1.sjc1.asianetcom.net (184.108.40.206) 195.939 ms 195.934 ms 196.090 ms
11 ip-202-147-50-250.asianetcom.net (220.127.116.11) 180.966 ms 181.058 ms 181.339 ms
12 dcx2-edge-01.inet.qwest.net (18.104.22.168) 342.431 ms 340.882 ms 338.639 ms
13 22.214.171.124 (126.96.36.199) 253.975 ms 253.407 ms 253.742 ms
14 vlan905.core5.iad2.rackspace.net (188.8.131.52) 253.468 ms 252.947 ms 253.639 ms
15 aggr301a-1-core5.iad2.rackspace.net (184.108.40.206) 253.767 ms 253.862 ms 253.010 ms
16 github.com (220.127.116.11) 254.104 ms 253.789 ms 253.638 ms
As shown in the NetworkMiner screenshot above, the Ethernet MAC address of the client's default gateway is ec:17:2f:15:23:b0, which indicates that the user's router was a device from TP-LINK. TP-LINK is a large provider of networking devices who make more than half of their sales within China. This supports the theory that the user was located in China.
As we've mentioned in a previous blog post, PcapNG files can contain a great deal of metadata. I therefore wrote a simple parser to extract all the juicy metadata that was available in the github.pcapng file that was anonymously uploaded to CloudShark. The metadata revealed that the user was running “64-bit Windows 7 Service Pack 1, build 7601” and sniffing with “Dumpcap 1.8.2 (SVN Rev 44520 from /trunk-1.8)”, which normally means that Wireshark 1.8.2 was being used (since it uses dumpcap for all packet capturing tasks).
The PcapNG file additionally contained a great deal of Name Resolution Blocks, i.e. cached DNS entries. Among these entries was an entry that mapped the IP 10.99.99.102 to the hostname “SHAOJU-IPAD.local”. So why was there a name resolution entry for an IP address that wasn't part of the capture file? Well, cached name resolution entries aren't properly cleaned from PcapNG files written with Wireshark 1.8.0 to 1.8.3 due to a vulnerability discovered by Laura Chappell.
||Is the user a victim of a real attack rather than having staged and recorded a MITM attack set up by himself?|
||Yes, this is most likely a REAL attack. I'd find it unlikely that a user trying to fake such a MITM attack would use an environment with six router hops between the client and MITM machine.|
||Is the user located in China?|
||Yes, I'm convinced that the PcapNG file was sniffed by Chen Shaoju, who lives in Wuxi, China according to his GitHub profile.|
||What can we say about the technology being used for the MITM?|
||The fact that the MITM machine was six hops away from the user indicates that the MITM is taking place at some fairly central position in China's internet infrastructure, as opposed to being done locally at the ISP. Additionally, the machine doing the MITM seems to be running Linux and having an initial IP TTL of 64 and a TCP Window Size of 5840.|
UPDATE (February 5) : Privacy / Anonymity
I did, however, contact Chen Shaoju before publishing this blog post. In fact, I even sent him an email as soon as we discovered that it was possible to identify him through the capture file's metadata. Here's what I wrote:
“I'm working on a blog post about the GitHub pcap file on CloudShark. My analysis indicates that you sniffed the traffic. Is it OK with you if I publish this blog post?Shaoju responded the same day and said it was OK.
If not, then I'd recommend that you remove the pcap file from couldshark in order to prevent others from identifying you via your pcap file.”
Once the text for the blog post was finished I also sent him a copy for verification before we published it online. Again, Shaoju responded swiftly and said that it looked good.
Also, I'm sure that Shaoju (@chenshaoju) can confirm that this blog post was published with his consent.
Finally, I'd like to add a few recommendations to users who wish to preserve their anonymity when sharing capture files:
- Filter out any traffic that isn't relevant to what you wish to share. CapLoader is a handy tool for selecting and filtering capture files based on flows or IP addresses.
- Save the capture file in the old libpcap (PCAP) format or convert your PcapNG file to PCAP format. This will remove the metadata available in Pcap-NG option fields.
- Anonymize frame headers, i.e Ethernet MAC addresses or even IP addresses (if needed), with a tool like: tcprewrite, Bit-Twist or TraceWrangler.
Posted by Erik Hjelmvik on Saturday, 02 February 2013 22:10:00 (UTC/GMT)
Version 1.1 of the super-fast PCAP parsing tool CapLoader is being released today. CapLoader is the ideal tool for digging through large volumes of PCAP files. Datasets in the GB and even TB order can be loaded into CapLoader to produce a clear view of all TCP and UDP flows. CapLoader also provides instantaneous access to the raw packets of each flow, which makes it a perfect preloader tool in order to select and export interesting data to other tools like NetworkMiner or Wireshark.
Five flows being extracted from Honeynet.org's SOTM 28 to Wireshark with CapLoader
New functionality in version 1.1
New features in version 1.1 of CapLoader are:
- PcapNG support
- Fast transcript of TCP and UDP flows (similar to Wireshark's ”Follow TCP Stream”)
- Better port agnostic protocol identification; more protocols and better precision (over 100 protocols and sub-protocols can now be identified, including Skype and the C&C protocol of Poison Ivy RAT)
- A “Hosts” tab containing a list of all transmitting hosts and information about open ports, operating system as well as Geo-IP localization (using GeoLite data created by MaxMind)
- Gzip compressed capture files can be opened directly with CapLoader
- Pcap files can be loaded directly from an URL
Flow transcript of Honeynet SOTM 28 pcap file day3.log
Free Trial Version
Another thing that is completely new with version 1.1 of CapLoader is that we now provide a free trial version for download. The CapLoader trial is free to use for anyone and we don't even require trial users to register their email addresses when downloading the software.
There are, of course, a few limitations in the trial version; such as no protocol identification, OS fingerprinting or GeoIP localization. There is also a limit as to how many gigabyte of data that can be loaded with the CapLoader trial at a time. This size limit is 500 GB, which should by far exceed what can be loaded with competing commercial software like Cascade Pilot and NetWitness Investigator.
The professional edition of CapLoader doesn't have any max PCAP limit whatsoever, which allows for terabytes of capture files to be loaded.
The CapLoader USB flash drive
Customers who have previously bought CapLoader 1.0 can upgrade to version 1.1 by downloading an update from our customer portal.
For more information about CapLoader please see our previous blog post Fast analysis of large pcap files with CapLoader
Posted by Erik Hjelmvik on Monday, 21 January 2013 11:45:00 (UTC/GMT)
Users of Wireshark 1.8.0 (or later) have most likely noticed that the default output file format has changed from libpcap (.pcap) to Pcap-NG (.pcapng). So what does this mean other than a longer file extension?
New Features in PcapNG
PcapNG is short for “PCAP Next Generation Dump File Format”. This new format isn't just an update of the old PCAP specification, but a total rewrite of the whole spec. Here are a few of the features that are available in the new PcapNG format:
- Traffic captured from multiple interfaces can be stored in a single file. This even works when the interfaces have different data link types, such as Ethernet, 802.11 (WiFi) and PPP.
- PcapNG trace files can be tagged with metadata info about what OS, hardware and sniffer application that was used to capture the traffic. Wireshark and dumpcap automatically tags generated PcapNG files this way.
- There is a long list of metadata attributes that can be stored about each interface. This list includes attributes like interface name, dropped packets and used capture filter.
- Text comments can be added and saved to individual frames. These comments, which are called “annotations”, are available in Wireshark and Tshark via the display filter named “pkt_comment”.
You can, for example, use tshark to list all annotations in a PcapNG file with the following command:
tshark -r dump.pcapng -T fields -e pkt_comment -R pkt_comment
You might also have seen improved timestamp resolution as a new feature of the PcapNG format. It is true that the PcapNG format allows for more precise timestamps compared to the microsecond resolution provided by the old PCAP format. This does, however, not mean that you will get nanosecond resolution in your capture files just because you switch to the PcapNG format. The default timestamp resolution in PcapNG files is still microseconds, and tools like Wireshark and dumpcap additionally only get microsecond resolution timestamps from the library they rely on for packet capturing (i.e. libpcap or WinPcap).
Many tools are not yet able to load PcapNG files, instead they'll spit out error messages like “bad dump file format”.
These error messages typically appear when a tool parses PCAP files with help of a libpcap version
prior to 1.1.0 (you can find the error message in
savefile.c from early 2010).
If you instead load a PcapNG file into NetworkMiner you'll currently get the following error message:
NetworkMiner 1.4.1 with error message while trying to open a PcapNG file
Convert PcapNG to PCAP
If you end up with a capture file in the PcapNG format, which you need to parse with a tool that does not yet support the “NG” format, then you'll have to convert it to the legacy PCAP format. You can use editcap to do the conversion like this:
editcap -F libpcap dump.pcapng dump.pcapHowever, if the PcapNG file contains traffic from multiple interfaces with different data link types then you'll get an error message saying:
editcap: Can't open or create dump.pcap: That file format doesn't support per-packet encapsulations
One way to convert such multi-encapsulation PcapNG files back to PCAP format is to first split the capture file into multiple PcapNG files, by using the display filters “frame.interface_id” or “frame.dlt”, and then convert each individual PcapNG file back to PCAP using editcap.
Another option is to submit your PcapNG file to the conversion service provided at pcapng.com. This free service only lets you convert file sizes up to 1MB, but it's really easy to use and you don't need to sign up or register to use it. Just choose a file and press “Convert to PCAP”, the site will then provide you with PCAP formatted capture files for each interface.
Sniff to PCAP
If you want to avoid conversion issues altogether then it's probably safest to stay with the good ol' PCAP format. But since dumpcap defaults to PcapNG nowadays you'll have to tell it to use the legacy PCAP format with the “-P” switch like this:
dumpcap -P -i eth0 -w dump.pcap
Posted by Erik Hjelmvik on Wednesday, 05 December 2012 12:15:00 (UTC/GMT)