NETRESEC Network Security Blog - Tag : Big Data


Analyzing 85 GB of PCAP in 2 hours

Hadoop photo by Robert Scoble

Lets say you've collected around 100 GB of PCAP files in a network monitoring installation. How would you approach the task of looking at the application layer data of a few of the captured sessions or flows?

For much smaller datasets, in the order of 100 MB, one would typically load the PCAP into Wireshark and perform ”Follow TCP Stream” on a few sessions to see what's going on. But loading gigabyte datasets into Wireshark doesn't scale very well, in fact Wireshark will typically run out of RAM and crash saying “Out Of Memory!” or just “Wireshark has stopped working”. Ulf Lamping explains why on the Wireshark Wiki:

“Wireshark uses memory to store packet meta data (e.g. conversation and fragmentation related data) and to display this info on the screen.
[...]
I need memory about ten times the actual capture file size”

The solution I'm proposing is to instead download the free version of CapLoader, load the PCAP files into CapLoader and perform ”Flow Transcript” on a few of the flows. So how long time would it take to do this on 100 GB of PCAP files? I did a quick test and loaded the 85 GB dataset from ISCX 2012 into CapLoader on an ordinary laptop computer. After just 1 hour 47 minutes all of the PCAP files from ISCX 2012 were loaded and indexed by CapLoader! Also, please note that datasets this large can be parsed in less than 30 minutes with a more powerful PC.

After having loaded all the PCAP files CapLoader presents a list of all the 2.066.653 indexed flows from the ISCX 2012 dataset. Right-clicking a UDP or TCP flow brings up a context menu that allows you to generate a “Flow Transcript”, this feature is basically the same thing as Wireshark's “Follow TCP Stream”.

Right-click a flow in CapLoader
CapLoader Flow Transcript
CapLoader's Flow Transcript View

You can, of course, always extract the frames from any flow directly to Wireshark if you aren't ready to abandon Wireshark's Follow TCP Stream just yet. A flow is extracted simply by selecting a flow in the list and then doing drag-and-drop from CapLoader's PCAP icon (at the top right) onto Wireshark.

Drag-and-Drop from CapLoader to Wireshark
Wireshark follow TCP stream

The fact that CapLoader parses and indexes large PCAP files very fast and that the analyst is provided with powerful tools, like the Transcript feature, to look at the raw packet data makes it possible to perform big-data network traffic analysis using an ordinary PC. This means that you do NOT need to upload your network traffic to the Cloud, or build a 100-machine cluster, in order to let a Hadoop instance parse though your multi-gigabyte packet captures. All you need is an ordinary PC and a copy of CapLoader.

For more information about CapLoader please have a look at our blog post highliting the new features in version 1.1 of CapLoader or browse through all our blog posts about CapLoader.

This blog post makes use of the UNB ISCX 2012 Intrusion Detection Evaluation Dataset, which is created by Ali Shiravi, Hadi Shiravi, and Mahbod Tavallaee from University of New Brunswick


UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to perform flow transcripts not only from the Flows tab, but also from the Services and Hosts tab. In these cases the transcript will be that of the first flow of the selected service or host.

Posted by Erik Hjelmvik on Thursday, 24 January 2013 12:20:00 (UTC/GMT)

Tags: #CapLoader #PCAP #gigabyte #GB #Wireshark #Follow TCP Stream #transcript #Big Data

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=13135F8


Fast analysis of large pcap files with CapLoader

CapLoader Logo

Are you working with large pcap files and need to see the “whole picture” while still being able to quickly drill down to individual packets for a TCP or UDP flow? Then this is your lucky day, since we at Netresec are releasing our new tool CapLoader today!

Here are the main features of CapLoader:

  • Fast loading of multi-gigabyte PCAP files (1 GB loads in less than 2 minutes on a standard PC and even faster on multi-core machines).
  • GUI presentation of all TCP and UDP flows in the loaded PCAP files.
  • Automatic identification of application layer protocols without relying on port numbers.
  • Extremely fast drill-down functionality to open packets from one or multiple selected flows.
  • Possibility to export packets from selected flows to a new PCAP file or directly open them in external tools like Wireshark and NetworkMiner.

CapLoader identifying Rootkit SSH backdoor on TCP 5001
CapLoader with files from Honeynet SOTM 28 loaded. The application layer protocol from the rootkit backdoor on TCP 5001 is automatically identified as "SSH".

The typical process of working with CapLoader is:

  1. Open one or multiple pcap files, typically by drag-and-dropping them onto the CapLoader GUI.
    CapLoader loading a pcap file with drag-and-drop
  2. Mark the flows of interest.
    CapLoader selecting flows / sessions
  3. Double click the PCAP icon to open the selected sessions in your default pcap parser (typically Wireshark) or better yet, do drag-and-drop from the PCAP icon to your favorite packet analyzer.
    CapLoader exporting packets to NetworkMiner

In short, CapLoader will significantly speed up the analysis process of large network captures while also empowering analysts with a unique protocol identification ability. We at Netresec see CapLoader as the perfect tool for everyone who want to perform analysis on “big data” network captures.

More information about CapLoader is available on caploader.com.

Posted by Erik Hjelmvik on Monday, 02 April 2012 19:55:00 (UTC/GMT)

Tags: #Netresec #CapLoader #Fast #Big Data #PCAP #Flow

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1246698

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)