NETRESEC Network Security Blog - Tag : flow


Don't Delete PCAP Files - Trim Them!

We are happy to release TrimPCAP today! TrimPCAP is a free open source tool that reduces the size of capture files in an intelligent way.

TrimPCAP

The retention period of a packet capture solution is typically limited by either legal requirements or available disk space. In the latter case the oldest capture files are simply removed when the storage starts getting full. This means that if there is a long ongoing session, such as a download of a large ISO file, streamed video or a reverse shell backdoor, then the start of this session will likely be removed.

I know from experience that it’s painful to analyze network traffic where the start of a session is missing. The most important and interesting stuff generally happens in the beginning of each session, such as the HTTP GET request for an ISO file. As an analyst you don’t need to look at all the other packets in that ISO download (unless you believe the ISO contains malware), it’s enough to see that there is a GET request for the file and the server responds with a “200 OK”.

CapLoader Transcript of ISO download Image: CapLoader transcript of an ISO download

If that download had been truncated, so that only the last few packets were remaining, then it would be really difficult to know what was being downloaded. The same is true also for other protocols, including proprietary C2 protocols used by botnets and other types of malware.


  TrimPCAP  

TrimPCAP is designed to overcome the issue with truncated sessions by removing data from the end of sessions rather than from the beginning. This also comes with a great bonus when it comes to saving on disk usage, since the majority of the bytes transferred across the Internet are made up of big sessions (a.k.a “Elephant Flows”). Thus, by trimming a PCAP file so that it only contains the first 100kB of each TCP and UDP session it’s possible to significantly reduce required storage for that data.

The following command reduces the PCAP dataset used in our Network Forensics Training from 2.25 GB to just 223 MB:

user@so$ python trimpcap.py 102400 /nsm/sensor_data/so-eth1/dailylogs/*/*
Trimming capture files to max 102400 bytes per flow.
Dataset reduced by 90.30% = 2181478771 bytes
user@so$

A maximum session size (or "flow cutoff") of 100kB enables trimpcap.py to reduce the required storage for that dataset to about 10% of its original size, which will significantly extend the maximum retention period.


Putting TrimPCAP Into Practice

Let’s assume that your organization currently has a maximum full content PCAP retention period of 10 days and that trimming sessions to 100kB reduces the required storage to 10%. TrimPCAP will then enable you to store 5 days with full session data plus 50 additional days with trimmed sessions using the same disk space as before!

TrimPCAP retention period extension

A slightly more advanced scheme would be to have multiple trim limits, such as trimming to 1MB after 3 days, 100kB after 6 days and 10kB after 30 days. Such a setup would probably extend your total retention period from 10 days to over 100 days.

An even more advanced trimming scheme is implemented in our packet capture agent PacketCache. PacketCache constantly trims its PCAP dataset because it is designed to use only 1 percent of a PC’s RAM to store observed packets, in case they are needed later on for incident response. PacketCache uses a trim limit which declines individually for each observed TCP and UDP session depending on when they were last observed.


Downloading TrimPCAP

TrimPCAP is open source software and is released under the GNU General Public License version 2 (GPLv2). You can download your own copy of TrimPCAP from the official TrimPCAP page:
https://www.netresec.com/?page=TrimPCAP

Happy trimming!

✂- - - - - - - - -

Posted by Erik Hjelmvik on Tuesday, 05 December 2017 12:40:00 (UTC/GMT)

Tags: #PCAP #python #flow

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=17CF81F


CapLoader 1.5 Released

CapLoader 1.5 Logo

We are today happy to announce the release of CapLoader 1.5. This new version of CapLoader parses pcap and pcap-ng files even faster than before and comes with new features, such as a built-in TCP stream reassembly engine, as well as support for Linux and macOS.

Support for ICMP Flows

CapLoader is designed to group packets together that belong to the same bi-directional flow, i.e. all UDP, TCP and SCTP packets with the same 5-tuple (regardless of direction) are considered being part of the same flow.

5-tuple
/fʌɪv ˈtjuːp(ə)l/
noun

A combination of source IP, destination IP, source port, destination port and transport protocol (TCP/UDP/SCTP) used to uniquely identify a flow or layer 4 session in computer networking.

The flow concept in CapLoader 1.5 has been extended to also include ICMP. Since there are no port numbers in the ICMP protocol CapLoader sets the source and destination port of ICMP flows to 0. The addition of ICMP in CapLoader also allows input filters and display filters like “icmp” to be leveraged.

Flows tab in CapLoader 1.5 with display filter BPF 'icmp'
Image: CapLoader 1.5 showing only ICMP flows due to display filter 'icmp'.

TCP Stream Reassembly

One of the foundations for making CapLoader a super fast tool for reading and filtering PCAP files is that it doesn’t attempt to reassemble TCP streams. This means that CapLoader’s Transcript view will show out-of-order segments in the order they were received and retransmitted segments will be displayed twice.

The basic concept has been to let other tools do the TCP reassembly, for example by exporting a PCAP for a flow from CapLoader to Wireshark or NetworkMiner.

The steps required to reassemble a TCP stream to disk with Wireshark are:

  1. Right-click a TCP packet in the TCP session of interest.
  2. Select “Follow > TCP Stream”.
  3. Choose direction in the first drop-down-list (client-to-server or server-to-client).
  4. Change format from “ASCII” to “Raw” in the next drop-down-menu.
  5. Press the “Save as...” button to save the reassembled TCP stream to disk.

Follow TCP Stream window in Wireshark

Unfortunately Wireshark fails to properly reassemble some TCP streams. As an example the current stable release of Wireshark (version 2.2.5) shows duplicate data in “Follow TCP Stream” when there are retransmissions with partially overlapping segments. We have also noticed some additional  bugs related to TCP stream reassembly in other recent releases of Wireshark. However, we’d like to stress that Wireshark does perform a correct reassembly of most TCP streams; it is only in some specific situations that Wireshark produces a broken reassembly. Unfortunately a minor bug like this can cause serious consequences, for example when the TCP stream is analyzed as part of a digital forensics investigation or when the extracted data is being used as input for further processing. We have therefore decided to include a TCP stream reassembly engine in CapLoader 1.5. The steps required to reassemble a TCP stream in CapLoader are:

  1. Double click a TCP flow of interest in the “Flows” tab to open a flow transcript.
  2. Click the “Save Client Byte Stream” or “Save Server Byte Stream” button to save the data stream for the desired direction to disk.

Transcript window in CapLoader 1-5

Extracting TCP streams from PCAP files this way not only ensures that the data stream is correctly reassembled, it is also both faster and simpler than having to pivot through Wireshark’s Follow TCP Stream feature.

PCAP Icon Context Menu

CapLoader 1.5 PCAP icon Save As...

The PCAP icon in CapLoader is designed to allow easy drag-and-drop operations in order to open a set of selected flows in an external packet analysis tool, such as Wireshark or NetworkMiner. Right-clicking this PCAP icon will bring up a context menu, which can be used to open a PCAP with the selected flows in an external tool or copy the PCAP to the clipboard. This context menu has been extended in CapLoader 1.5 to also include a “Save As” option. Previous versions of CapLoader required the user to drag-and-drop from the PCAP icon to a folder in order to save filtered PCAP data to disk.

Faster Parsing with Protocol Identification

CapLoader can identify over 100 different application layer protocols, including HTTP, SSL, SSH, RTP, RTCP and SOCKS, without relying on port numbers. The protocol identification has previously slowed down the analysis quite a bit, which has caused many users to disable this powerful feature. This new release of of CapLoader comes with an improved implementation of the port-independent protocol identification feature, which enables PCAP files to be loaded twice as fast as before with the “Identify protocols” feature enabled.

Works in Linux and macOS

One major improvement in CapLoader 1.5 is that this release is compatible with the Mono framework, which makes CapLoader platform independent. This means that you can now run CapLoader on your Mac or Linux machine if you have Mono installed. Please refer to our previous blog posts about how to run NetworkMiner in various flavors of Linux and macOS to find out how to install Mono on your computer. You will, however, notice a performance hit when running CapLoader under Mono instead of using Windows since the Mono framework isn't yet as fast as Microsoft's .NET Framework.

CapLoader 1.5 running in Linux with Mono
Image: CapLoader 1.5 running in Linux (Xubuntu).

Credits

We’d like to thank Sooraj for reporting a bug in the “Open With” context menu of CapLoader’s PCAP icon. This bug has been fixed in CapLoader 1.5 and Sooraj has been awarded an official “PCAP or it didn’t happen” t-shirt for reporting the bug.

PCAP or it didn't happen t-shirt
Image: PCAP or it didn't happen t-shirt

Have a look at our Bug Bounty Program if you also wanna get a PCAP t-shirt!

Downloading CapLoader 1.5

Everything mentioned in this blog post, except for the protocol identification feature, is available in our free trial version of CapLoader. To try it out, simply grab a copy here:
https://www.netresec.com/?page=CapLoader#trial (no registration needed)

All paying customers with an older version of CapLoader can download a free update to version 1.5 from our customer portal.

Posted by Erik Hjelmvik on Tuesday, 07 March 2017 09:11:00 (UTC/GMT)

Tags: #CapLoader #PCAP #pcap-ng #Follow TCP Stream #Wireshark #Linux #Mac #RTP #SOCKS #Mono #Flow

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1738EBA


Keyword Search in PCAP files

Sherlock Holmes and Magnifying Glass via Inside Croydon A new function in the free version of CapLoader 1.2 is the "Find Keyword" feature. This keyword search functionality makes it possible to seek large capture files for a string or byte pattern super fast!

You might say, so what? PCAP string search can already be done with tools like tcpflow, ngrep and even Wireshark; what's the benefit of adding yet another tool to this list? One benefit is that CapLoader doesn't just give you the packet or content that matched the keyword, it will instead extract the whole TCP or UDP flow that contained the match. CapLoader also supports many different encodings, which is demonstrated in this blog post.

Here are a few quick wins with CapLoader's keyword search feature:

  • Track User-Agent - Search for a specific user agent string to extract all the HTTP traffic from a particular browser or malware.
  • Track Domain Name - Search for a particular domain name to get all DNS lookups as well as web traffic relating to that domain (including HTTP "referer" field matches).
  • Extract Messages - Search for a keyword in e-mail or chat traffic to get the whole e-mail or conversation, not just the single packet that matched.
  • Extract Files - Search for a unique string or byte sequence in a file (such as a piece of malware) to enable extraction of the complete file transfer.

EXAMPLE: DigitalCorpora M57

As an example, let's search the digital corpora file net-2009-12-06-11:59.pcap (149 MB) for the keyword "immortal". Follow these steps in order to veify our analysis using the free edition of CapLoader.

  1. Start CapLoader and select File -> Open URL, enter:
    http://digitalcorpora.org/corp/nps/scenarios/2009-m57-patents/net/net-2009-12-06-11:59.pcap.gz
  2. Edit -> Find Keyword (or Ctrl+F), enter "immortal" CapLoader Find Keyword Form
  3. Click the "Find and Select All Matching Flows" button
  4. One TCP flow is now selected (Flow_ID 5469, 192.168.1.104:2592 -> 192.168.1.1:25) CapLoader with one selected flow
  5. Right click the selected flow (ID 5469) and select "Flow Transcript"
CapLoader Flow Transcript of SMTP email attachment

CapLoader transcript of SMTP email flow

Looks as if an email has been sent with an attachment named "microscope1.jpg". However, the string "immortal" cannot be seen anywhere in the transcript view. The match that CapLoader found was actually in the contents of the attachment, which has been base64 encoded in the SMTP transfer in accordance with RFC 2045 (MIME).

The email attachment can easily be extracted from the PCAP file using NetworkMiner. However, to keep things transparent, let's just do a simple manual verification of the matched data. The first three lines of the email attachment are:

/9j/4AAQSkZJRgABAQEAkACQAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB
AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEB
AQEBAQEBAQEBAQEBAQEBAQEBAQFwYXNzd29yZD1pbW1vcnRhbAEBAQEBAQEBAQEBAQH/wAAR
Decoding this with base64 gives us:
0000000: ffd8 ffe0 0010 4a46 4946 0001 0101 0090 ......JFIF......
0000010: 0090 0000 ffdb 0043 0001 0101 0101 0101 .......C........
0000020: 0101 0101 0101 0101 0101 0101 0101 0101 ................
0000030: 0101 0101 0101 0101 0101 0101 0101 0101 ................
0000040: 0101 0101 0101 0101 0101 0101 0101 0101 ................
0000050: 0101 0101 0101 0101 01ff db00 4301 0101 ............C...
0000060: 0101 0101 0101 0101 0101 0101 0101 0101 ................
0000070: 0101 0101 0101 0101 0101 0101 0101 0101 ................
0000080: 7061 7373 776f 7264 3d69 6d6d 6f72 7461 password=immorta
0000090: 6c01 0101 0101 0101 0101 0101 0101 ffc0 l...............

Tools like ngrep, tcpflow and Wireshark won't find any match for the string "immortal" since they don't support searching in base64 encoded data. CapLoader, on the other hand, supports lots of encodings.

Supported Text Encodings

CapLoader currently supports fast searching of text strings in any of the following encodings:

  • ASCII
  • Base64 (used in email attachments and HTTP POST's)
  • DNS label encoding (RFC 1035)
  • HTML
  • Quoted Printable (used in body of email messages)
  • Unicode
  • URL encoding
  • UTF8

CapLoader also supports several local character sets, including the following code pages:

  • 437 MS-DOS Latin US
  • 850 MS-DOS Latin 1
  • 932 Japanese
  • 936 Simplified Chinese
  • 949 Korean
  • 1251 Windows Cyrillic (Slavic)
  • 1256 Windows Arabic

Having all these encodings also makes it possible to search network traffic for words like хакер, القراصنة, ハッカー, 黑客 or 해커.

The Art of War by Sun Tzu

Getting CapLoader

CapLoader is a commercial tool that also comes in a free trial edition. The search feature is available in both versions, so feel free to download CapLoader and try it your self!

CapLoader is available from the following URL:
http://www.netresec.com/?page=CapLoader

Posted by Erik Hjelmvik on Wednesday, 02 April 2014 13:15:00 (UTC/GMT)

Tags: #search #find #keyword #flow #stream #PCAP #SMTP #transcript #free #network

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1447A3D


CapLoader 1.1 Released

CapLoader Logo Version 1.1 of the super-fast PCAP parsing tool CapLoader is being released today. CapLoader is the ideal tool for digging through large volumes of PCAP files. Datasets in the GB and even TB order can be loaded into CapLoader to produce a clear view of all TCP and UDP flows. CapLoader also provides instantaneous access to the raw packets of each flow, which makes it a perfect preloader tool in order to select and export interesting data to other tools like NetworkMiner or Wireshark.

Drag-and-Drop PCAP from CapLoader to Wireshark
Five flows being extracted from Honeynet.org's SOTM 28 to Wireshark with CapLoader

New functionality in version 1.1

New features in version 1.1 of CapLoader are:

  • PcapNG support
  • Fast transcript of TCP and UDP flows (similar to Wireshark's ”Follow TCP Stream”)
  • Better port agnostic protocol identification; more protocols and better precision (over 100 protocols and sub-protocols can now be identified, including Skype and the C&C protocol of Poison Ivy RAT)
  • A “Hosts” tab containing a list of all transmitting hosts and information about open ports, operating system as well as Geo-IP localization (using GeoLite data created by MaxMind)
  • Gzip compressed capture files can be opened directly with CapLoader
  • Pcap files can be loaded directly from an URL

CapLoader Flow Transcript aka Follow TCP Stream
Flow transcript of Honeynet SOTM 28 pcap file day3.log

Free Trial Version

Another thing that is completely new with version 1.1 of CapLoader is that we now provide a free trial version for download. The CapLoader trial is free to use for anyone and we don't even require trial users to register their email addresses when downloading the software.

There are, of course, a few limitations in the trial version; such as no protocol identification, OS fingerprinting or GeoIP localization. There is also a limit as to how many gigabyte of data that can be loaded with the CapLoader trial at a time. This size limit is 500 GB, which should by far exceed what can be loaded with competing commercial software like Cascade Pilot and NetWitness Investigator.

The professional edition of CapLoader doesn't have any max PCAP limit whatsoever, which allows for terabytes of capture files to be loaded.

CapLoader with TCP and UDP flows view
CapLoader's Flows view showing TCP and UDP flows

CapLoader with Hosts view
CapLoader's Hosts view showing identified hosts on the network

Getting CapLoader

The trial version of CapLoader can be downloaded from the CapLoader product page. The professional edition of CapLoader can be bought at our Purchase CapLoader page.

CapLoader USB flash drive
The CapLoader USB flash drive

Customers who have previously bought CapLoader 1.0 can upgrade to version 1.1 by downloading an update from our customer portal.

For more information about CapLoader please see our previous blog post Fast analysis of large pcap files with CapLoader

Posted by Erik Hjelmvik on Monday, 21 January 2013 11:45:00 (UTC/GMT)

Tags: #CapLoader #PcapNG #PCAP #GB #gigabyte #capture #flow #transcript #free

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1310CE5


CapLoader Video Tutorial

CapLoader Logo

Below is a short video tutorial showing some of the cool features in CapLoader 1.0.

The functionality showed in the video includes:

  • Loading multiple pcap files into a single flow view
  • Port Independent Protocol Identification (PIPI)
  • Fast extraction of packets related to one or several flows
  • Exporting packets to Wireshark and NetworkMiner
  • Drag-and-dropping packets to Wireshark
  • Selecting a flow based on an IDS alert from Snort
  • Extracting packets from a selected flow to a new pcap file

The video can also be seen on YouTube at the following URI:
http://youtu.be/n1Ir9Hedca4?hd=1

The three pcap files loaded in the video tutorial are from the DFRWS 2009 Challenge.

Enjoy!

Posted by Erik Hjelmvik on Monday, 30 April 2012 14:35:00 (UTC/GMT)

Tags: #CapLoader #Video #Pcap #Wireshark #NetworkMiner #Flow #TCP #Extract #Fast

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=124DA49


Fast analysis of large pcap files with CapLoader

CapLoader Logo

Are you working with large pcap files and need to see the “whole picture” while still being able to quickly drill down to individual packets for a TCP or UDP flow? Then this is your lucky day, since we at Netresec are releasing our new tool CapLoader today!

Here are the main features of CapLoader:

  • Fast loading of multi-gigabyte PCAP files (1 GB loads in less than 2 minutes on a standard PC and even faster on multi-core machines).
  • GUI presentation of all TCP and UDP flows in the loaded PCAP files.
  • Automatic identification of application layer protocols without relying on port numbers.
  • Extremely fast drill-down functionality to open packets from one or multiple selected flows.
  • Possibility to export packets from selected flows to a new PCAP file or directly open them in external tools like Wireshark and NetworkMiner.

CapLoader identifying Rootkit SSH backdoor on TCP 5001
CapLoader with files from Honeynet SOTM 28 loaded. The application layer protocol from the rootkit backdoor on TCP 5001 is automatically identified as "SSH".

The typical process of working with CapLoader is:

  1. Open one or multiple pcap files, typically by drag-and-dropping them onto the CapLoader GUI.
    CapLoader loading a pcap file with drag-and-drop
  2. Mark the flows of interest.
    CapLoader selecting flows / sessions
  3. Double click the PCAP icon to open the selected sessions in your default pcap parser (typically Wireshark) or better yet, do drag-and-drop from the PCAP icon to your favorite packet analyzer.
    CapLoader exporting packets to NetworkMiner

In short, CapLoader will significantly speed up the analysis process of large network captures while also empowering analysts with a unique protocol identification ability. We at Netresec see CapLoader as the perfect tool for everyone who want to perform analysis on “big data” network captures.

More information about CapLoader is available on caploader.com.

Posted by Erik Hjelmvik on Monday, 02 April 2012 19:55:00 (UTC/GMT)

Tags: #Netresec #CapLoader #Fast #Big Data #PCAP #Flow

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1246698

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)