NETRESEC Network Security Blog - Tag : Capture

PCAP or it didn't happen

The phrase "PCAP or it didn't happen" is often used in the network security field when someone want proof that an attack or compromise has taken place. One such example is the recent OpenSSL heartbleed vulnerability, where some claim that the vulnerability was known and exploited even before it was discovered by Google's Neel Mehta and Codenomicon.

PCAP or it didn't happen pwnie, original by Nina on
Image: PCAP or it didn't happen pwnie, original by Nina

After the Heartbleed security advisory was published, EFF tweeted:

"Anyone reproduced observations of #Heartbleed attacks from 2013?"
and Liam Randall (of Bro fame) tweeted:
"If someone finds historical exploits of #Heartbleed I hope they can report it. Lot's of sites mining now."

Liam Randall (@Hectaman) tweeting about historical Heartbleed searches Heartbleed

It is unfortunately not possible to identify Heartbleed attacks by analyzing log files, as stated by the following Q&A from the website:

Can I detect if someone has exploited this against me?

Exploitation of this bug does not leave any trace of anything abnormal happening to the logs.

Additionally, IDS  signatures  for detecting the Heartbleed attacks weren't available until after implementations of the exploit code were being actively used in the wild.

Hence, the only reliable way of detecting early heartbleed attacks (i.e. prior to April 7) is to analyze old captured network traffic from before April 7. In order to do this you should have had a full packet capture running, which was configured to capture and store all your traffic. Unfortunately many companies and organizations haven't yet realized the value that historical packet captures can provide.

Why Full Packet Capture Matters

Some argue that just storing netflow data is enough in order to do incident response. However, detecting events like the heartbleed attack is impossible to do with netflow since you need to verify the contents of the network traffic.

Not only is retaining historical full packet captures useful in order to detect attacks that have taken place in the past, it is also extremely valuable to have in order to do any of the following:

  • IDS Verification
    Investigate IDS alerts to see if they were false positives or real attacks.

  • Post Exploitation Analysis
    Analyze network traffic from a compromise to see what the attacker did after hacking into a system.

  • Exfiltration Analysis
    Assess what intellectual property that has been exfiltrated by an external attacker or insider.

  • Network Forensics
    Perform forensic analysis of a suspect's network traffic by extracting files, emails, chat messages, images etc.

Setting up a Full Packet Capture

netsniff-ng logo

The first step, when deploying a full packet capture (FPC) solution, is to install a network tap or configure a monitor port in order to get a copy of all packets going in and out from your networks. Then simply sniff the network traffic with a tool like dumpcap or netsniff-ng. Another alternative is to deploy a whole network security monitoring (NSM) infrastructure, preferably by installing the SecurityOnion Linux distro.

A network sniffer will eventually run out of disk, unless captured network traffic is written to disk in a rung buffer manner (use "-b files" switch in dumpcap) or there is a scheduled job in place to remove the oldest capture files. SecurityOnion, for example, normally runs its "cleandisk" cronjob when disk utilization reaches 90%.

The ratio between disk space and utilized bandwidth becomes the maximum retention period for full packet data. We recommend having a full packet capture retention period of at least 7 days, but many companies and organizations are able to store several month's worth of network traffic (disk is cheap).

Big Data PCAP Analysis

Okay, you've got a PCAP store with multiple terabytes of data. Then what? How do you go about analyzing such large volumes of captured full content network traffic? Well, tasks like indexing and analyzing PCAP data are complex matters than are beyond the scope of this blog post. We've covered the big data PCAP analysis topic in previous  blog posts, and there is more to come. However, capturing the packets to disk is a crucial first step in order to utilize the powers of network forensics. Or as the saying goes “PCAP or it didn't happen”.

UPDATE 2015-06-02

We now have T-shirts with "PCAP or it didn't happen" print for sale!

PCAP or it didn't happen T-shirt

Posted by Erik Hjelmvik on Thursday, 01 May 2014 21:45:00 (UTC/GMT)

Tags: #capture #sniffer #IDS #forensics

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

CapLoader 1.1 Released

CapLoader Logo Version 1.1 of the super-fast PCAP parsing tool CapLoader is being released today. CapLoader is the ideal tool for digging through large volumes of PCAP files. Datasets in the GB and even TB order can be loaded into CapLoader to produce a clear view of all TCP and UDP flows. CapLoader also provides instantaneous access to the raw packets of each flow, which makes it a perfect preloader tool in order to select and export interesting data to other tools like NetworkMiner or Wireshark.

Drag-and-Drop PCAP from CapLoader to Wirehsark
Five flows being extracted from's SOTM 28 to Wireshark with CapLoader

New functionality in version 1.1

New features in version 1.1 of CapLoader are:

  • PcapNG support
  • Fast transcript of TCP and UDP flows (similar to Wireshark's ”Follow TCP Stream”)
  • Better port agnostic protocol identification; more protocols and better precision (over 100 protocols and sub-protocols can now be identified, including Skype and the C&C protocol of Poison Ivy RAT)
  • A “Hosts” tab containing a list of all transmitting hosts and information about open ports, operating system as well as Geo-IP localization (using GeoLite data created by MaxMind)
  • Gzip compressed capture files can be opened directly with CapLoader
  • Pcap files can be loaded directly from an URL

CapLoader Flow Transcript aka Follow TCP Stream
Flow transcript of Honeynet SOTM 28 pcap file day3.log

Free Trial Version

Another thing that is completely new with version 1.1 of CapLoader is that we now provide a free trial version for download. The CapLoader trial is free to use for anyone and we don't even require trial users to register their email addresses when downloading the software.

There are, of course, a few limitations in the trial version; such as no protocol identification, OS fingerprinting or GeoIP localization. There is also a limit as to how many gigabyte of data that can be loaded with the CapLoader trial at a time. This size limit is 500 GB, which should by far exceed what can be loaded with competing commercial software like Cascade Pilot and NetWitness Investigator.

The professional edition of CapLoader doesn't have any max PCAP limit whatsoever, which allows for terabytes of capture files to be loaded.

CapLoader with TCP and UDP flows view
CapLoader's Flows view showing TCP and UDP flows

CapLoader with Hosts view
CapLoader's Hosts view showing identified hosts on the network

Getting CapLoader

The trial version of CapLoader can be downloaded from the CapLoader product page. The professional edition of CapLoader can be bought at our Purchase CapLoader page.

CapLoader USB flash drive
The CapLoader USB flash drive

Customers who have previously bought CapLoader 1.0 can upgrade to version 1.1 by downloading an update from our customer portal.

For more information about CapLoader please see our previous blog post Fast analysis of large pcap files with CapLoader

Posted by Erik Hjelmvik on Monday, 21 January 2013 11:45:00 (UTC/GMT)

Tags: #CapLoader #PcapNG #PCAP #GB #gigabyte #capture #flow #transcript #free

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

SCADA Network Forensics with IEC-104


A great way to enable digital forensics of control system networks is to implement network security monitoring. Captured network traffic is a great source for evidence when analyzing an attackers steps as he attempts to hack a SCADA system. The newly added support for the IEC-104 protocol in NetworkMiner also allows investigators and incident responders to see what commands the attacker sent to the control system.

We at Netresec recently announced the release of NetworkMiner 1.4, which comes with a parser for the SCADA protocol IEC 60870-5-104 (aka IEC-104). Bringing this Industrial Control System (ICS) protocol into NetworkMiner is a first step to support forensics of compromised ICS networks. The traffic from ICS networks does, of course, need to be captured (sniffed) in order to support network forensics; we are strong supporters of such network monitoring for ICS networks (read our “Monitor those Control System Networks” blog post for more details).

Why monitor ICS networks?

Computer forensics typically involves performing forensic analysis of hard disks. Disk forensics is very effective when analyzing a hard drive from a PC (like an operator workstation), but far more complicated when it is an embedded device like a PLC or RTU that is to be analyzed.

In regard to what was believed to be a hacked SCADA system at a water facility in Illinois, David Marcus from McAfee said:

“My gut tells me that there is greater targeting and wider compromise than we know about. Why? Again, my instincts tell me that there is a lack of cyber forensics and response procedures at most of these facilities. If you do not have cyber forensic capabilities, it is hard to know if you have a cyber intrusion.”

Even though the hack was later shown to just be just a false alarm, David’s point about lacking capabilities for digital forensics and incident response for this type of critical infrastructure still holds true.

Joe Weiss also commented on the same story saying:

“We don't know how many other SCADA systems have been compromised because they don't really have cyber forensics.”

As Joe and David say, the ability to perform digital forensics in SCADA systems is truly lacking today. Our propose with this blog post is to inform control system operators that forensic data/evidence can be easily collected from ICS / SCADA systems by implementing a simple solution for network monitoring with full packet capture.

How to monitor ICS networks

The SCADA network diagram below has been sectioned into multiple security zones according to the zoning principle published by Jens Z, Iiro and me at CIRED 2009 (our zones align nicely with ISA-99 security Levels by the way).

SCADA Network with security zones

The purple octagons represent interconnections between zones. Each such interconnection should be secured with perimeter protection, typically by a firewall, but we additionally argue that all network traffic passing through should be captured and stored as pcap files. Storing all network traffic this way makes it possible to perform network forensics on the network traffic after an intrusion is believed to have taken place.

We recommend a very simple setup, where a network tap is used to provide a copy of all traffic to a sniffer. An acceptable alternative to buying a network tap is to configure a monitor / SPAN port on a switch (see our sniffing tutorial “Intercepting Network Traffic” for more details on how to choose sniffing hardware).

Connection of network tap and sniffer

Our recommended solution for the sniffer is to install FreeBSD with dumpcap (part of the net/tshark ports package). An even easier solution is to install Doug BurksSecurity Onion, which is a Linux distro built especially for network security monitoring. More about configuring a sniffer can be found in our second sniffing tutorial titled “Dumping Network Traffic to Disk”.

Analyzing captured IEC 104 traffic

Let’s assume the file 090813_diverse.pcap from pcapr contains network traffic from a suspected security breach at a hydro-power plant. Let’s also assume that parameter 4821 (i.e. IOA 4821 in IEC-104 language) controls the floodgates of the plant’s dam, where setting a value greater than 0% for this parameter would mean opening the floodgates.

By loading the pcap file into NetworkMiner and selecting the “parameters” tab we can see a nice log of all IEC-104 communication.

NetworkMiner 1.4.1 with Parameters tab

NOTE: We’ve hidden several fields (like IP, port, time etc) in the screenshot above in order to make it fit.

The following timeline can be extracted from the list of events provided by NetworkMiner:

  • Frame 154 - The attacker sends command to set IOA 4821 to 50.354%
  • Frame 156 - The RTU confirms the request
  • Frame 162 - The RTU reports that the requested command has been successfully completed, i.e. floodgates are now open!

Open dam gates by David Baird

More ICS protocols

Would you like to see more ICS protocols in NetworkMiner? We’d be happy to implement protocols like DNP3, MODBUS, ICCP, Siemens S7, IEC 61850, etc. if you can provide us with captured network traffic! Please send an email to info[at] if you are interested!

Posted by Erik Hjelmvik on Thursday, 30 August 2012 12:03:00 (UTC/GMT)

Tags: #Forensics #ICS #SCADA #control system #Network #Sniff #Capture #Monitor #pcap

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Sniffing Tutorial part 1 - Intercepting Network Traffic

A cow's nose

This blog post is the first part of a two-part tutorial that shows how to sniff network traffic. This first part covers how to intercept the traffic, i.e. how to get the packets to arrive to your network card. The second part covers how to best capture the network traffic to disk once you've managed to have them sent to your network card.

I divide the field of network forensics and network security monitoring into two disciplines; capturing (a.k.a. sniffing) the traffic and analyzing the captured traffic. Analyzing the network traffic can be done on many different levels, from the novice's level where IP addresses and layer 7 protocols are listed to the expert who understands all protocol layers and can investigate traffic packet-by-packet. Network traffic analysis is a field where there is always something more to learn and where nobody, not even an expert like Richard Bejtlich, knows everything. Capturing traffic, on the other hand, is more like a craftsmanship; it takes a while to learn how to do it properly, but there is not much new to learn once you've gotten the hang of it. Being able to capture network traffic reliably is, however, essential in order to perform network traffic analysis or network forensics, which is why I decided to write this sniffing tutorial.

Many “n00bs” fire up Wireshark on their own PC expecting to be able to sniff all traffic passing through an Ethernet network. But the 90's are long gone, and all hubs have been replaced by switches, so your NIC nowadays only gets broadcast packets and packets addressed to your NIC. So if you wanna capture traffic from other hosts on the network you somehow need to force that traffic to passes by your NIC. I will here explain some of the most usual ways to achieve this.

Use a legacy “dumb” hub

4 port Netgear Ethernet hub

The good ol' dumb hubs (a.k.a. active hubs) are hard to find these days, but if you're fortunate enough to own one of those old jewels then make sure to hold on to it. You might also get lucky and find a used hub on ebay. These beauties repeat all Ethernet frames arriving at one port to all the other ports on the hub, which is just what we want in order to sniff all the traffic. Just place the hub at a choke-point in your network, for example between your firewall and the rest of the network, and connect your sniffer PC directly to the hub (see my ASCII image below).

Hub in choke-point

One downside of using a hub is that all these extra packets being sent on all the hub's ports increase the risk of packet collisions, i.e. when two nodes on a shared media network are transmitting at the same time. Every time a collision occurs on an Ethernet network the nodes on the shared media segment will pause for a random time before they continue sending data, this functionality is called CSMA/CD. For TCP based protocols such collisions result in lots of ugly retransmissions, while the data in collided UDP packets never will reach their destination since UDP doesn't support retransmissions.

Configure a monitor port on a managed switch

Network switches

A much better solution than using a hub is to configure a monitor port on a managed switch, from which you wish to tap some traffic. Monitor ports have different names depending on the vendor, Cisco call theirs SPAN ports, others use the term “port mirroring”. A monitor port is typically configured so that it copies all sent and received frames for a particular port to a monitor port, which is where you connect your sniffer. You can normally also configure the switch to copy traffic from all ports to the monitor port, but this can cause duplicate packets since each packet will be copied both when coming in on one port as well as when it goes out on another port. It is therefore usually better to monitor only the uplink port if you wish to monitor all hosts connected to a switch.

The major downside of sniffing traffic of a monitor port is that the primary functionality of a switch is to forward traffic from the sender to the receiver, not to send copies of that traffic to a monitor port. This means that if there is a high load on the switch it will prioritize sending the received frames to the recipient over copying them to the monitor port. You therefore run the risk of not always getting all transmitted packets to your monitor port, which might be acceptable in some situations but isn't acceptable in a network forensics investigation. Finding out if your monitor port has missed some packets isn't easy, but Charles Smutz explains how this can be done in his blog post “Flushing out Leaky Taps”.

Network Tap

The most reliable way to sniff traffic is to use a network tap. A network tap is a “bump-in-the-wire” device designed only to copy traffic passing through it to a monitor port. You typically insert a network tap inline between two nodes in a network, such as between your firewall and your first switch. A good network tap will cost you at least 1.000 EUR, but those are money well spent if you wanna be able to perform reliable network monitoring.

If you're planning to buy a network tap I recommend getting a link aggregation tap, which can merge uplink and downlink traffic to one single monitor port. This way you don't need two network interfaces on your sniffer PC. Both major tap vendors Net Optics and VSS Monitoring sell aggregation taps.

A downside of using a network tap compared to configuring a monitor port is that there will be a short downtime on the network while connecting and disconnecting the tap. I therefore recommend to install network taps when you have a service window. And when you have a tap connected at a good choke-point in your network I recommend that you leave it there after you're done, since you never know when you might need to come back and sniff some more.

ARP poisoning

A technique commonly used by hackers and penetration testers for getting hold of traffic in a switched environment is to use ARP poisoning. Basically ARP poisoning is a technique where two hosts on a network are tricked into sending packets destined for each other to a sniffer machine on the network. I'm not gonna describe this method in any more detail than so since ARP poisoning is a technique that I don't recommend using. Tools used for performing ARP poisoning are hacker tools like Cain & Abel and Ettercap. Another similar technique that I also don't recommend using is overflowing the CAM table.

Still not bored?

If you wanna learn even more about setting up monitor ports and sniffing traffic I recommend that you take a look at Betty DuBois' presentation titled “BU-11: SPAN/Mirror/Monitor vs. Taps: When should I use what, and why should I care?” from Sharkfest '09. The good people of LoveMyTool have provided a video recording of Betty's talk at

There is also a great list of various network tapping options over at the Argus website.

Continue reading part 2 of this sniffing tutorial, where I cover how to dump the traffic to disk.

Posted by Erik Hjelmvik on Friday, 11 March 2011 16:52:00 (UTC/GMT)

Tags: #Sniff #Capture #Monitor

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:


NETRESEC on Twitter

Follow @netresec on twitter:


Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)