HowTo handle PcapNG files

Users of Wireshark 1.8.0 (or later) have most likely noticed that the default output file format has changed from libpcap (.pcap) to Pcap-NG (.pcapng). So what does this mean other than a longer file extension?

New Features in PcapNG

PcapNG is short for “PCAP Next Generation Dump File Format”. This new format isn't just an update of the old PCAP specification, but a total rewrite of the whole spec. Here are a few of the features that are available in the new PcapNG format:

• Traffic captured from multiple interfaces can be stored in a single file. This even works when the interfaces have different data link types, such as Ethernet, 802.11 (WiFi) and PPP.
• PcapNG trace files can be tagged with metadata info about what OS, hardware and sniffer application that was used to capture the traffic. Wireshark and dumpcap automatically tags generated PcapNG files this way.
• There is a long list of metadata attributes that can be stored about each interface. This list includes attributes like interface name, dropped packets and used capture filter.
• Text comments can be added and saved to individual frames. These comments, which are called “annotations”, are available in Wireshark and Tshark via the display filter named “pkt_comment”.

You can, for example, use tshark to list all annotations in a PcapNG file with the following command:

tshark -r dump.pcapng -T fields -e pkt_comment -R pkt_comment

You might also have seen improved timestamp resolution as a new feature of the PcapNG format. It is true that the PcapNG format allows for more precise timestamps compared to the microsecond resolution provided by the old PCAP format. This does, however, not mean that you will get nanosecond resolution in your capture files just because you switch to the PcapNG format. The default timestamp resolution in PcapNG files is still microseconds, and tools like Wireshark and dumpcap additionally only get microsecond resolution timestamps from the library they rely on for packet capturing (i.e. libpcap or WinPcap).

Compatibility Issues

Many tools are not yet able to load PcapNG files, instead they'll spit out error messages like “bad dump file format”. These error messages typically appear when a tool parses PCAP files with help of a libpcap version prior to 1.1.0 (you can find the error message in savefile.c from early 2010). If you instead load a PcapNG file into NetworkMiner you'll currently get the following error message:

NetworkMiner 1.4.1 with error message while trying to open a PcapNG file

Convert PcapNG to PCAP

If you end up with a capture file in the PcapNG format, which you need to parse with a tool that does not yet support the “NG” format, then you'll have to convert it to the legacy PCAP format. You can use editcap to do the conversion like this:

editcap -F libpcap dump.pcapng dump.pcap

editcap: Can't open or create dump.pcap: That file format doesn't support per-packet encapsulations

One way to convert such multi-encapsulation PcapNG files back to PCAP format is to first split the capture file into multiple PcapNG files, by using the display filters “frame.interface_id” or “frame.dlt”, and then convert each individual PcapNG file back to PCAP using editcap.

Another option is to submit your PcapNG file to the conversion service provided at pcapng.com. This free service only lets you convert file sizes up to 1MB, but it's really easy to use and you don't need to sign up or register to use it. Just choose a file and press “Convert to PCAP”, the site will then provide you with PCAP formatted capture files for each interface.

Sniff to PCAP

If you want to avoid conversion issues altogether then it's probably safest to stay with the good ol' PCAP format. But since dumpcap defaults to PcapNG nowadays you'll have to tell it to use the legacy PCAP format with the “-P” switch like this:

dumpcap -P -i eth0 -w dump.pcap

Posted by Erik Hjelmvik on Wednesday, 05 December 2012 12:15:00 (UTC/GMT)

Tags: #PcapNG #PCAP #convert #editcap #tshark #NetworkMiner #dumpcap

Pcap-over-IP in NetworkMiner

Version 1.1 of NetworkMiner is soon to be released by us at Netresec. I would therefore like to give you a sneak preview of a simple yet very useful feature that we've added. We call this new feature “Pcap-over-IP”, which is a name originally coined by Packet Forensics.

With Pcap-over-IP you can have NetworkMiner read a pcap file (or libpcap formatted data in general) or over a TCP socket instead of getting it via the file system. The easiest way to send a pcap file over a TCP socket is to pipe a pcap file to netcat like this:

# cat sniffed.pcap | nc 192.168.1.20 57012

In this example I'd be running NetworkMiner on a PC with IP 192.168.1.20 and have Pcap‑over‑IP listening to TCP port 57012. NetworkMiner will save the received packets to disk as well as parse and display the contents of the packets in the GUI when receiving the Pcap‑over‑IP stream.

Pcap-over-IP also allows me to do live network sniffing with dumpcap from my local Windows machine and pipe the captured packets to NetworkMiner via a TCP socket, using Netcat for Windows like this:

C:\Program Files\Wireshark>dumpcap -i 4 -P -w - | C:\Tools\Netcat\nc.exe 127.0.0.1 57012

Note that the “-w -” switch tells dumpcap to push the raw libpcap formated data to standard output (stdout) rather than saving it to a pcap file.

The reason for using dumcap to perform the live sniffing rather than using the built in packet capturing functionality of NetworkMiner is that dumpcap is an extremely reliably tool when it comes to capturing packets. So by sniffing with dumpcap instead of NetworkMiner you minimize the risk of dropping some packets.

I can also use Pcap-over-IP to capture network traffic from a remote PC or device. I can, for example use tcpdump to sniff traffic on the external interface of my Linux-based firewall and push it to an analyst station like this:

# tcpdump -i eth1 -s 0 -U -w - | nc 192.168.1.20 57012

I can also perform remote WiFi sniffing with dumpcap or tcpdump from a Linux machine and send the sniffed packets to NetworkMiner with netcat like this:

# iwconfig wlan0 mode monitor
# iwconfig wlan0 channel 4
# dumpcap -i wlan0 -P -w - | nc 192.168.1.20 57012

It is even possible to receive multiple PCAP streams simultaneously with NetworkMiner. This way I could have 14 dumpcap or tcpdump processes sniffing each individual IEEE 802.11 channel, while monitoring all the captured traffic in real-time with a single instance of NetworkMiner. However, note that this would require 14 sniffer computers or a single sniffer machine with 14 WiFi cards.

SSL encryption

Don't like sending your pcap files in cleartext over the network? That's just fine, we've also implemented support for SSL/TLS encryption in NetworkMiner. You can use the great multipurpose relay tool socat to read your pcap file and have it encrypted with SSL while transiting the network like this:

# socat GOPEN:sniffed.pcap SSL:192.168.1.20:57013,verify=0

You can also use socat when doing live sniffing like this:

# tcpdump -i br0 -s 0 -U -w - | socat - SSL:192.168.1.20:57013,verify=0

Warning: Always make sure you don't sniff your own Pcap-over-IP stream when sending packets to NetworkMiner. You will otherwise construct a feedback loop, which will fill up the tubes. If you need to sniff the same interface as you are using to perform the Pcap‑over‑IP transfer, then make sure to use BPF to filter out the port number used for your Pcap‑over‑IP transfer like this:

# tcpdump -i ppp0 -U -w - not port 57012 | nc 192.168.1.20 57012

UPDATE June 16, 2014

With the release of NetworkMiner 1.6 we've made the PCAP‑over‑IP functionality available in the free open source edition of NetworkMiner. We have also integrated PCAP‑over‑IP into NetworkMinerCLI, i.e. the command line version of NetworkMiner Professional.

Posted by Erik Hjelmvik on Wednesday, 07 September 2011 09:22:00 (UTC/GMT)

Tags: #Netresec #Pcap-over-IP #Pcap #netcat #tcpdump #dumpcap #TCP #SSL #TLS