HowTo handle PcapNG files

Users of Wireshark 1.8.0 (or later) have most likely noticed that the default output file format has changed from libpcap (.pcap) to Pcap-NG (.pcapng). So what does this mean other than a longer file extension?

New Features in PcapNG

PcapNG is short for “PCAP Next Generation Dump File Format”. This new format isn't just an update of the old PCAP specification, but a total rewrite of the whole spec. Here are a few of the features that are available in the new PcapNG format:

• Traffic captured from multiple interfaces can be stored in a single file. This even works when the interfaces have different data link types, such as Ethernet, 802.11 (WiFi) and PPP.
• PcapNG trace files can be tagged with metadata info about what OS, hardware and sniffer application that was used to capture the traffic. Wireshark and dumpcap automatically tags generated PcapNG files this way.
• There is a long list of metadata attributes that can be stored about each interface. This list includes attributes like interface name, dropped packets and used capture filter.
• Text comments can be added and saved to individual frames. These comments, which are called “annotations”, are available in Wireshark and Tshark via the display filter named “pkt_comment”.

You can, for example, use tshark to list all annotations in a PcapNG file with the following command:

tshark -r dump.pcapng -T fields -e pkt_comment -R pkt_comment

You might also have seen improved timestamp resolution as a new feature of the PcapNG format. It is true that the PcapNG format allows for more precise timestamps compared to the microsecond resolution provided by the old PCAP format. This does, however, not mean that you will get nanosecond resolution in your capture files just because you switch to the PcapNG format. The default timestamp resolution in PcapNG files is still microseconds, and tools like Wireshark and dumpcap additionally only get microsecond resolution timestamps from the library they rely on for packet capturing (i.e. libpcap or WinPcap).

Compatibility Issues

Many tools are not yet able to load PcapNG files, instead they'll spit out error messages like “bad dump file format”. These error messages typically appear when a tool parses PCAP files with help of a libpcap version prior to 1.1.0 (you can find the error message in savefile.c from early 2010). If you instead load a PcapNG file into NetworkMiner you'll currently get the following error message:

NetworkMiner 1.4.1 with error message while trying to open a PcapNG file

Convert PcapNG to PCAP

If you end up with a capture file in the PcapNG format, which you need to parse with a tool that does not yet support the “NG” format, then you'll have to convert it to the legacy PCAP format. You can use editcap to do the conversion like this:

editcap -F libpcap dump.pcapng dump.pcap

editcap: Can't open or create dump.pcap: That file format doesn't support per-packet encapsulations

One way to convert such multi-encapsulation PcapNG files back to PCAP format is to first split the capture file into multiple PcapNG files, by using the display filters “frame.interface_id” or “frame.dlt”, and then convert each individual PcapNG file back to PCAP using editcap.

Another option is to submit your PcapNG file to the conversion service provided at pcapng.com. This free service only lets you convert file sizes up to 1MB, but it's really easy to use and you don't need to sign up or register to use it. Just choose a file and press “Convert to PCAP”, the site will then provide you with PCAP formatted capture files for each interface.

Sniff to PCAP

If you want to avoid conversion issues altogether then it's probably safest to stay with the good ol' PCAP format. But since dumpcap defaults to PcapNG nowadays you'll have to tell it to use the legacy PCAP format with the “-P” switch like this:

dumpcap -P -i eth0 -w dump.pcap

Posted by Erik Hjelmvik on Wednesday, 05 December 2012 12:15:00 (UTC/GMT)

Tags: #PcapNG #PCAP #convert #editcap #tshark #NetworkMiner #dumpcap

Pcap-over-IP in NetworkMiner

Version 1.1 of NetworkMiner is soon to be released by us at Netresec. I would therefore like to give you a sneak preview of a simple yet very useful feature that we've added. We call this new feature “Pcap-over-IP”, which is a name originally coined by Packet Forensics.

With Pcap-over-IP you can have NetworkMiner read a pcap file (or libpcap formatted data in general) or over a TCP socket instead of getting it via the file system. The easiest way to send a pcap file over a TCP socket is to pipe a pcap file to netcat like this:

# cat sniffed.pcap | nc 192.168.1.20 57012

In this example I'd be running NetworkMiner on a PC with IP 192.168.1.20 and have Pcap‑over‑IP listening to TCP port 57012. NetworkMiner will save the received packets to disk as well as parse and display the contents of the packets in the GUI when receiving the Pcap‑over‑IP stream.

Pcap-over-IP also allows me to do live network sniffing with dumpcap from my local Windows machine and pipe the captured packets to NetworkMiner via a TCP socket, using Netcat for Windows like this:

C:\Program Files\Wireshark>dumpcap -i 4 -P -w - | C:\Tools\Netcat\nc.exe 127.0.0.1 57012

Note that the “-w -” switch tells dumpcap to push the raw libpcap formated data to standard output (stdout) rather than saving it to a pcap file.

The reason for using dumcap to perform the live sniffing rather than using the built in packet capturing functionality of NetworkMiner is that dumpcap is an extremely reliably tool when it comes to capturing packets. So by sniffing with dumpcap instead of NetworkMiner you minimize the risk of dropping some packets.

I can also use Pcap-over-IP to capture network traffic from a remote PC or device. I can, for example use tcpdump to sniff traffic on the external interface of my Linux-based firewall and push it to an analyst station like this:

# tcpdump -i eth1 -s 0 -U -w - | nc 192.168.1.20 57012

I can also perform remote WiFi sniffing with dumpcap or tcpdump from a Linux machine and send the sniffed packets to NetworkMiner with netcat like this:

# iwconfig wlan0 mode monitor
# iwconfig wlan0 channel 4
# dumpcap -i wlan0 -P -w - | nc 192.168.1.20 57012

It is even possible to receive multiple PCAP streams simultaneously with NetworkMiner. This way I could have 14 dumpcap or tcpdump processes sniffing each individual IEEE 802.11 channel, while monitoring all the captured traffic in real-time with a single instance of NetworkMiner. However, note that this would require 14 sniffer computers or a single sniffer machine with 14 WiFi cards.

SSL encryption

Don't like sending your pcap files in cleartext over the network? That's just fine, we've also implemented support for SSL/TLS encryption in NetworkMiner. You can use the great multipurpose relay tool socat to read your pcap file and have it encrypted with SSL while transiting the network like this:

# socat GOPEN:sniffed.pcap SSL:192.168.1.20:57013,verify=0

You can also use socat when doing live sniffing like this:

# tcpdump -i br0 -s 0 -U -w - | socat - SSL:192.168.1.20:57013,verify=0

Warning: Always make sure you don't sniff your own Pcap-over-IP stream when sending packets to NetworkMiner. You will otherwise construct a feedback loop, which will fill up the tubes. If you need to sniff the same interface as you are using to perform the Pcap‑over‑IP transfer, then make sure to use BPF to filter out the port number used for your Pcap‑over‑IP transfer like this:

# tcpdump -i ppp0 -U -w - not port 57012 | nc 192.168.1.20 57012

UPDATE June 16, 2014

With the release of NetworkMiner 1.6 we've made the PCAP‑over‑IP functionality available in the free open source edition of NetworkMiner. We have also integrated PCAP‑over‑IP into NetworkMinerCLI, i.e. the command line version of NetworkMiner Professional.

Posted by Erik Hjelmvik on Wednesday, 07 September 2011 09:22:00 (UTC/GMT)

Tags: #Netresec #Pcap-over-IP #Pcap #tcpdump #dumpcap #TCP #SSL #TLS

Sniffing Tutorial part 2 - Dumping Network Traffic to Disk

This blog post is the second part of a two-part tutorial that shows how to sniff network traffic. The first part covered how to intercept the traffic, i.e. how to get the packets to arrive to your network card. This second part covers how to best capture the network traffic to disk once you've managed to have them sent to your network card.

The first thing I recommend you to do before you start sniffing is to ensure that your sniffer computer stays quiet, so that you don't pollute the sniffed traffic with packets from your sniffer computer. How to do this varies between Windows, Linux and FreeBSD, so I will briefly cover how to silence all three OS's.

Silencing Windows

Windows is normally quite chatty on the network, but there is a simple trick that makes it shut up without disconnecting it from the network. Simply bring up the properties window for your network adapter and un-check all clients, services and protocols from the list of used items.

Silencing Linux

I don't know any good way to disable the TCP/IP stack in Linux similar to what we just did to Windows. But by disabling ARP and not giving the interface any IP address the risk of leaking packets can be reduced significantly. To achieve this you'll also need to set the interface to promiscuous mode and disable IPv6. Here's an example of how this can be done in Debian/Ubuntu:

# ifconfig eth0 -arp up
# ifconfig eth0 promisc
# ip addr del 192.168.3.4/24 dev eth0
# ifconfig eth0 del fe80::1234:ddff:fe3f:1337/64

auto eth0
  iface eth0 inet manual
  up ifconfig $IFACE -arp up
  up ip link set $IFACE promisc on
  down ip link set $IFACE promisc off
  down ifconfig $IFACE down

For more details on shuting up your Linux machine I suggest you take a look at the “Undetectable Sniffing On Ethernet” blog post on the Ask Apache blog. I also recommend reading the Ubuntu instructions at the NetworkConfiguration wiki page for Security Onion.

Silencing FreeBSD

One of the many great things about FreeBSD is that it is really easy to configure a network interface to be silent. All you need to do is to issue this single command:

# ifconfig em0 monitor up

Read Richard Bejtlich's blog post “Ifconfig Monitor Option” for more info about this command.

Sniff with dumpcap

The most common software used for sniffing traffic is undoubtedly Wireshark, other common tools are tshark and tcpdump. All these three tools parse the packets as they come in and show data about the sniffed packets in the GUI or on the command line. This isn't desirable when doing network forensics since these application could use up too much resources on parsing the incoming packets, which during traffic peaks can lead to packets being dropped. There might also be bugs in the packet parsing code that can cause the application to crash completely, there have been plenty of such bugs in Wireshark for example.

A much more reliable way to sniff traffic is therefore to use an application that is designed to reliably dump traffic to disk, without trying to parse or analyze the contents of the sniffed frames. One such application is dumpcap, which is a command line tool developed by the Wireshark crew. In fact, as of version 1.4.0 of Wireshark both tshark and Wireshark run dumpcap in the background rather than doing the sniffing themselves. Another great thing about dumpcap is that it is available on Windows as well as on Linux and FreeBSD. My recommendation is to run it under FreeBSD since I consider it to be more reliable than the other operating systems. In order to get dumcap you need to download either Wirehsark or tshark since dumpcap doesn't have an installation package of its own.

Dumpcap supports capture filtering with BPF, but I recommend that you sniff ALL packets if you are authorized to do so and have enough disk space. It is much better to do the filtering later on in the analysis phase. The feature I like best in dumpcap is the ring buffer functionality, where it will continue writing incoming frames to a new pcap file once the old one has reached a specific size limit. In order to have dumpcap write sniffed frames to disk and roll over to a new file when it reaches 100.000 kB you'd run this command:

# dumpcap -i em0 -w wiretap.pcap -b filesize:100000

When you've dumped the traffic to disk is when the fun starts, i.e. analyzing the pcap files. You can read more analyzing pcap files in my blog posts about the TCP/IP Weapons School Sample Lab, DFRWS 2009 Challenge, Webmail Information Leakage and Command-line forensics.

Posted by Erik Hjelmvik on Friday, 11 March 2011 16:56:00 (UTC/GMT)

Tags: #dumpcap #Wireshark #tshark #dump #pcap