NETRESEC Network Security Blog - Tag : BPF


CapLoader 1.6 Released

CapLoader 1.6

CapLoader is designed to simplify complex tasks, such as digging through gigabytes of PCAP data looking for traffic that sticks out or shouldn’t be there. Improved usability has therefore been the primary goal, when developing CapLoader 1.6, in order to help our users do their work even more efficiently than before.

Some of the new features in CapLoader 1.6 are:

  • Context aware selection and filter suggestions when right-clicking a flow, session or host.
  • Support for IPv6 addresses in the BPF syntax for Input Filter as well as Display Filter.
  • Flows that are inactive for more than 60 minutes are considered closed. This timeout is configurable in Tools > Settings.


Latency Measurements

CapLoader 1.6 also introduces a new column in the Flows tab labeled “Initial_RTT”, which shows the Round Trip Time (RTT) measured during the start of a session. The RTT is defined as “the time it takes for a signal to be sent plus the time it takes for an acknowledgment of that signal to be received”. RTT is often called “ping time” because the ping utility computes the RTT by sending ICMP echo requests and measuring the delay until a reply is received.

Initial RTT in CapLoader Flows Tab
Image: CapLoader 1.6 showing ICMP and TCP round trip times.

But using a PCAP file to measure the RTT between two hosts isn’t as straight forward as one might think. One complicating factor is that the PCAP might be generated by the client, server or by any device in between. If we know that the sniffing point is at the client then things are simple, because we can then use the delta-time between an ICMP echo request and the returning ICMP echo response as RTT. In lack of ping traffic the same thing can be achieved with TCP by measuring the time between a SYN and the returning SYN+ACK packet. However, consider the situation when the sniffer is located somewhere between the client and server. The previously mentioned method would then ignore the latency between the client and sniffer, the delta-time will therefore only show the RTT between the sniffer and the server.

This problem is best solved by calculating the Initial RTT (iRTT) as the delta-time between the SYN packet and the final ACK packet in a TCP three-way handshake, as shown here:

Initial Round Trip Time in PCAP Explained
Image: Initial RTT is the total time of the black/bold packet traversal paths.

Jasper Bongertz does a great job of explaining why and how to use the iRTT in his blog post “Determining TCP Initial Round Trip Time”, so I will not cover it in any more detail here. However, keep in mind that iRTT can only be calculated this way for TCP sessions. CapLoader therefore falls back on measuring the delta time between the first packet in each direction when it comes to transport protocols like UDP and ICMP.


Exclusive Features Not Available in the Free Trail

The new features mentioned so far are all available in the free 30 day CapLoader trial, which can be downloaded from our CapLoader product page (no registration required). But we’ve also added features that are only available in the commercial/professional edition of CapLoader. One such exclusive feature is the matching of hostnames against the Cisco Umbrella top 1 million domain list. CapLoader already had a feature for matching domain names against the Alexa top 1 million list, so the addition of the Umbrella list might seem redundant. But it’s actually not, the two lists are compiled using different data sources and therefore complement each other (see our blog post “Domain Whitelist Benchmark: Alexa vs Umbrella” for more details). Also, the Umbrella list contains subdomains (such as www.google.com, safebrowsing.google.com and accounts.google.com) while the Alexa list only contains main domains (like “google.com”). CapLoader can therefore do more fine-granular domain matching with the Umbrella list (requiring a full match of the Umbrella domain), while the Alexa list enables a more rough “catch ‘em all” approach (allowing *.google.com to be matched).

CapLoader Hosts tab with ASN, Alexa and Umbrella details

CapLoader 1.6 also comes with an ASN lookup feature, which presents the autonomous system number (ASN) and organization name for IPv4 and IPv6 addresses in a PCAP file (see image above). The ASN lookup is built using the GeoLite database created by MaxMind. The information gained from the MaxMind ASN database is also used to provide intelligent display filter CIDR suggestions in the context menu that pops up when right-clicking a flow, service or host.

CapLoader Flows tab with context menu for Apply as Display Filter
Image: Context menu suggests Display Filter BPF “net 104.84.152.0/17” based on the server IP in the right-clicked flow.

Users who have previously purchased a license for CapLoader can download a free update to version 1.6 from our customer portal.


Credits and T-shirts

We’d like to thank Christian Reusch for suggesting the Initial RTT feature and Daan from the Dutch Ministry of Defence for suggesting the ASN lookup feature. We’d also like to thank David Billa, Ran Tohar Braun and Stephen Bell for discovering and reporting bugs in CapLoader which now have been fixed. These three guys have received a “PCAP or it didn’t happen” t-shirt as promised in our Bug Bounty Program.

Got a t-shirt for crashing CapLoader

If you too wanna express your view of outlandish cyber attack claims without evidence, then please feel free to send us your bug reports and get rewarded with a “PCAP or it didn’t happen” t-shirt!

Posted by Erik Hjelmvik on Monday, 09 October 2017 08:12:00 (UTC/GMT)

Tags: #CapLoader #free #IPv6 #BPF #CIDR #PCAP #Umbrella #Alexa

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=17ABA35


BPF is your Friend

CapLoader BPF

CapLoader comes with support for Berkeley Packet Filter (BPF), which makes it possible to filter network traffic based on IP addresses, protocols and port numbers without using external tools. Being able to filter captured network traffic is crucial when analyzing large sets of PCAP files as well as in order to hunt down compromised hosts with Rinse Repeat Intrusion Detection.

There are two ways to apply filters with BPF in CapLoader; you can either apply an input filter before loading your PCAPs, or you can apply a display filter after the capture files have been loaded.


Input Filter

The fastest way to filter a large set of PCAP files with CapLoader is to enter an Input Filter before loading the capture files. Having an input filter will speed up the loading time significantly, since CapLoader will not need to analyze packets and flows that don't match the BPF syntax. The downside is that you will need to know beforehand what filter you want to use. In order to apply a changed input filter you need to reload the loaded PCAP files (pressing F5 will do this for you).

CapLoader with input filter “tcp port 443”
Image: CapLoader with input filter “tcp port 443”

Display Filter

CapLoader supports display filters in order to allow filters to be changed on the fly, without having to reload the PCAP files. As the name implies, display filters affect what flows/services/hosts that are displayed in CapLoader. A changed display filter does not require the dataset to be reloaded, but it does require the GUI to update the visible flows. This GUI update will be somewhat slower compared to when setting an input filter.

CapLoader with display filter “host 94.23.23.39”
Image: CapLoader with display filter “host 94.23.23.39”

BPF Syntax

CapLoader's BPF implementation does not support the full BPF syntax. In fact, only the most central primitives are implemented, which are:

host <IP address>Flows to or from the specified IPv4 or IPv6 address
net <CIDR> Flows to or from the specified IP network, uses CIDR notation
port <port>Flows to or from the specified port number
ip6Flows using IPv6 addresses
ipFlows using IPv4 addresses
tcpTCP flows
udpUDP flows
sctpSCTP flows

More complex filter expressions can be built up by using the words and, or, not and parentheses to combine primitives. Here are some examples:

  • host 8.8.8.8 and udp port 53
  • net 199.16.156.0/22 and port 80
  • (port 80 or port 443) and not host 192.168.0.1

For all boolean algebra geeks out there we can confirm that our BPF implementation gives and precedence over or, which means that the last example above would give a different result if the parentheses were removed.


Keeping it Short

Steve McCanne gave a keynote presentation at SharkFest 2011, where he talked about how he created BPF. Steve's work was guided by Van Jacobson, who challenged him to make the BPF syntax human friendly rather than requiring the user to type a clunky filtering syntax. We've adopted this thinking and therefore allow filters like these:

  • 10.1.1.3
    Flows to or from IP address 10.1.1.3. Translates to “ip host 10.1.1.3”

  • 128.3/16
    Flows to or from the 128.3.0.0/16 network. Translates to “ip net 128.3.0.0/16”

  • port 53
    Flows to or from TCP, UDP or SCTP port 53.


Try it for Free!

We've made the BPF implementation available even in the free version of CapLoader. You don't need to register to get the free version; just download, extract and run. The tool is portable, so you won't even have to install it. Visit https://www.netresec.com/?page=CapLoader to grab a copy and start filtering!


UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to apply Display Filters not only to the Flows tab, but also to the Services and Hosts tab.

Posted by Erik Hjelmvik on Monday, 30 November 2015 08:15:00 (UTC/GMT)

Tags: #CapLoader #BPF #PCAP #Berkeley Packet Filter #filter #IP #port #CIDR

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=15B2F44


CapLoader 1.3 Released

CapLoader Logo

A new version of our heavy-duty PCAP parser tool CapLoader is now available. There are many new features and improvements in this release, such as the ability to filter flows with BPF, domain name extraction via passive DNS parser and matching of domain names against a local white list.


Filtering with BPF

The main focus in the work behind CapLoader 1.3 has been to fully support the Rinse-Repeat Intrusion Detection methodology. We've done this by improving the filtering capabilities in CapLoader. For starters, we've added an input filter, which can be used to specify IP addresses, IP networks, protocols or port numbers to be parsed or ignored. The input filter uses the Berkeley Packet Filter (BPF) syntax, and is designed to run really fast. So if you wanna analyze only HTTP traffic you can simply write “port 80” as your input filter to have CapLoader only parse and display flows going to or from port 80. We have also added a display filter, which unlike Wireshark also uses BPF. Thus, once a set of flows is loaded one can easily apply different display filters, like “host 194.9.94.80” or “net 192.168.1.0/24”, to apply different views on the parsed data.

CapLoader BPF Input Filter and Display Filter
Image: CapLoader with input filter "port 80 or port 443" and display filter "not net 74.125.0.0/16".

The main differences between the input filter and display filter are:

  • Input filter is much faster than the display filter, so if you know beforehand what ports, protocols or IP addresses you are interested in then make sure to apply them as an input filter. You will notice a delay when applying a display filter to a view of 10.000 flows or more.
  • In order to apply a new input filter CapLoader has to reload all the opened PCAP files (which is done by pressing F5). Modifying display filters, on the other hand, only requires you to press Enter or hit the “Apply” button.
  • Previously applied display filters are accessible in a drop-down menu in the GUI, but no history is kept of previous input filters.


NetFlow + DNS == true

The “Flows” view in CapLoader gives a great overview of all TCP, UDP and SCTP flows in the loaded PCAP files. However, it is usually not obvious to an analyst what every IP address is used for. We have therefore added a DNS parser to CapLoader, so that all DNS packets can be parsed in order to map IP addresses to domain names. The extracted domain names are displayed for each flow, which is very useful when performing Rinse-Repeat analysis in order to quickly remove “known good servers” from the analysis.


Leveraging the Alexa top 1M list

As we've show in in our previous blog post “DNS whitelisting in NetworkMiner”, using a list of popular domain names as a whitelist can be an effective method for finding malware. We often use this approach in order to quickly remove lots of known good servers when doing Rinse-Repeat analysis in large datasets.

Therefore, just as we did for NetworkMiner 1.5, CapLoader now includes Alexa's list of the 1 million most popular domain names on the Internet. All domain names, parsed from DNS traffic, are checked against the Alexa list. Domains listed in the whitelist are shown in CapLoader's “Server_Alexa_Domian” column. This makes it very easy to sort on this column in order to remove (hide) all flows going to “normal” servers on the Internet. After removing all those flows, what you're left with is pretty much just:

  • Local traffic (not sent over the Internet)
  • Outgoing traffic to either new or obscure domains

Manually going through the remaining flows can be very rewarding, as it can reveal C2 traffic from malware that has not yet been detected by traditional security products like anti-virus or IDS.

Flows in CapLoader with DNS parsing and Alexa lookup
Image: CapLoader with malicious flow to 1.web-counter[.]info (Miuref/Boaxxe Trojan) singled out due to missing Alexa match.

Many new features in CapLoader 1.3

The new features highlighted above are far from the only additions made to CapLoader 1.3. Here is a more complete list of improvements in this release:

  • Support for “Select Flows in PCAP” to extract and select 5-tuples from a PCAP-file. This can be a Snort PCAP with packets that have triggered IDS signatures. This way you can easily extract the whole TCP or UDP flow for each signature match, instead of just trying to make sense of one single packet per alert.
  • Improved packet carver functionality to better carve IP, TCP and UPD packets from any file. This includes memory dumps as well as proprietary and obscure packet capture formats.
  • Support for SCTP flows.
  • DNS parser.
  • Alexa top 1M matching.
  • Input filter and display filter with BPF syntax.
  • Flow Producer-Consumer-Ratio PCR.
  • Flow Transcript can be opened simply by double-clicking a flow.
  • Find form updated with option to hide non-matching flows instead of just selecting the flows that matched the keyword search criteria.
  • New flow transcript encoding with IP TTL, TCP flags and sequence numbers to support analysis of Man-on-the-Side attacks.
  • Faster loading of previously opened files, MD5 hashes don't need to be recalculated.
  • A selected set of flows in the GUI can be inverted simply by right-clicking the flow list and selecting “Invert Selection” or by hitting Ctrl+I.


Downloading CapLoader 1.3

All these new features, except for the Alexa lookup of domain names, are available in our free trial version of CapLoader. So to try out these new features in CapLoader, simply grab a trial download here:
https://www.netresec.com/?page=CapLoader#trial (no registration needed)

All paying customers with an older version of CapLoader can grab a free update for version 1.3 at our customer portal.

Posted by Erik Hjelmvik on Monday, 28 September 2015 07:30:00 (UTC/GMT)

Tags: #CapLoader #BPF #Berkeley Packet Filter #Rinse-Repeat #DNS #Alexa #PCAP #NetFlow #C2

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=15914E3

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)