NETRESEC Network Security Blog - Tag : filter


NetworkMiner 2.0 Released

NetworkMiner 2.0

I'm proud to announce the release of NetworkMiner 2.0 today! There are several longed-for features that are part of this major release, such as:

  • SMB/CIFS parser now supports file extraction from SMB write operations.
  • Added parser for SMB2 protocol (read and write).
  • Additional IEC-104 commands implemented.
  • Added Modbus/TCP parser (as requested by attendees at 4SICS 2014).
  • Improved SMTP parser.
  • Improved FTP parser.
  • Improved DNS parser.
  • GUI flickering is heavily reduced when loading PCAP files or doing live sniffing.
  • Extraction of web server favicon images (shown in Hosts tab).
  • Added "Keyword filter" to several tabs (see more details below).
NetworkMiner 2.0 showing hosts in nitroba.pcap
Image: NetworkMiner 2.0 showing hosts in nitroba.pcap from Digital Corpora

For those not familiar with NetworkMiner, here's a short summary:

NetworkMiner is a Network Forensic Analysis Tool (NFAT) for Windows (but also works in Linux / Mac OS X / FreeBSD). NetworkMiner can be used as a passive network sniffer/packet capturing tool in order to detect operating systems, sessions, hostnames, open ports etc. without putting any traffic on the network. NetworkMiner can also parse PCAP files for off-line analysis and to regenerate/reassemble transmitted files and certificates from PCAP files.

NetworkMiner has, since the first release in 2007, become a popular tool among incident response teams as well as law enforcement. NetworkMiner is today used by companies and organizations all over the world.


Keyword Filtering

Users of NetworkMiner sometimes run into a needle-in-the-haystack problem when trying to find some specific entry in the Files, Parameters or DNS tab. The technique most analysts have been using so far is to sort the data based a column of interest (by clicking the column header) and then scroll down to the row they are looking for. This method isn't optimal, which is why several users have requested support for keyword filtering. I've personally received several of these requests when teaching my Networks Forensics class.

I'm therefore happy to say that version 2.0 of NetworkMiner has a built-in filtering capability for the data displayed in the following tabs:

  • Files
  • Parameters
  • DNS
  • Messages
  • Sessions

Parameters tab with filter user-agent
Image: Parameters tab with filter “user-agent” (case insensitive)

The filter allows rows to be filtered based on one or several keywords. The entered keywords are matched against all text in all columns of the tab. A drop-down menu in the filter bar allows the analyst to chose “Exact Phrase”, “All Words” or “Any Word” as search criteria for the entered keywords.


More Data in the Parameters Tab

We have previously held back on what data we add to the Parameters tab. However, now with the filtering feature in place, we decided to add a lot more information to the Parameters list. Some of the new parameter types available in version 2.0 are:

  • HTTP request methods
  • HTTP URI's
  • HTTP response status codes
  • HTTP headers
  • SMB Tree Connect AndX Request (attempts to connect to a named file share)
  • SMB NT Create AndX Request (mapping of filename to file handle ID)
  • SMB2 Connect Requests (attempts to connect to a named file share)
  • SMB2 File ID (mapping of filename to file handle ID)
  • SMB2 file timestamps (Created, Modified and Accessed).

SMB2 file timestamps shown in Parameters tab
Image: SMB2 file timestamps shown in Parameters tab


A Warning to Malware Analysts

NetworkMiner has previously appended the “.octet-stream” or “x-msdos-program” extension to all binary files being downloaded over HTTP (since those are the MIME types used for Windows executables). As of version 2.0, however, files named {something}.exe will not be renamed this way. This means that there is now a risk of accidental execution of such files, for example if the user right-clicks an .exe file in NetworkMiner and selects “Open File”.

NetworkMiner with extracted .exe file

If you analyze PCAP files that might contain malware, then our recommendation is to perform the analysis on some other operating system than Windows. NetworkMiner runs fine on Linux as well as Mac OS X.


NetworkMiner Professional

On top of the updates provided in the free version of NetworkMiner we have added a few additional useful features to NetworkMiner Professional.

The new features in the Pro version include:

  • Advanced OS fingerprinting. Identifies a great range of operating systems and device types (including Apple iOS, Android and many others) based on DHCP traffic.
  • Web Browsing Analysis. A new GUI tab called "Browsers" has been added, which shows what URLs each unique browser has visited. More details on this feature will be released shortly in a separate blog post.
  • User Settings. Settings in GUI can now be saved to make them persistent between executions.
  • Better Export Logs. The CSV format used for exporting data has been improved, we have also added support for XML formatted data export.

Andoid, Apple iOS, Mac OS and Windows detected in captured DHCP traffic from a WiFi network
Image: Andoid, Apple iOS, Mac OS and Windows detected in captured DHCP traffic from a WiFi network.

Defang Executables

Remember the warning about .exe files downloaded over HTTP no longer get the “.octet-stream” extension? To counter the risk of accidental execution of malware we've added a defang feature to NetworkMiner Professional. When enabled, this feature will rename files like “malware.exe” to “malware.exe_” in order to prevent execution. It is not only .exe files being renamed. At the moment the following file extensions are defanged by appending “_” to the extension:

exe, bat, msi, vb, vbe, vbs, pif, com, scr, jar, cmd, js, jse, ps1 and psc1

The defang feature renames files regardless if they were transmitted over HTTP, FTP, TFTP, SMB, SMB2, SMTP or any other protocol supported by NetworkMiner.

Settings window in NetworkMiner Professional

Please note that the defang feature is turned off by default. So if you wanna protect yourself from accidental execution, make sure to enable this feature in the NetworkMiner Settings (available under Tools > Settings).


Upgrade Path for Professional Users

We always provide free minor version upgrades of our software. However, please note that version 2.0 is a major version release, which require a new license to be purchased. Customers with a license for NetworkMiner Professional 1.x can get a 20% discount when purchasing version 2.0 by entering their current license number into the “Additional Information” field of the NetworkMiner Professional order form.


Credits

There are many people who have provided feedback and ideas for improvements that are now part of NetworkMiner 2.0. I would especially like to thank Eric Kollmann (author of Satori) for his impressive ability to find bugs in beta releases. I would also like to thank Ulf Skoglund, Dmitry Shchemelev, @xredumb, Sabin and Andrew Brant for their feedback on improvements in DNS, TCP, HTTP and SMTP parsing. Finally, I wanna give a shout-out to Steffen Thorkildsen for spawning the idea for a browser tracking feature.

Posted by Erik Hjelmvik on Tuesday, 09 February 2016 11:38:00 (UTC/GMT)

Tags: #NetworkMiner #SMB #Modbus #filter #favicon #SMTP #email

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=162BA65


BPF is your Friend

CapLoader BPF

CapLoader comes with support for Berkeley Packet Filter (BPF), which makes it possible to filter network traffic based on IP addresses, protocols and port numbers without using external tools. Being able to filter captured network traffic is crucial when analyzing large sets of PCAP files as well as in order to hunt down compromised hosts with Rinse Repeat Intrusion Detection.

There are two ways to apply filters with BPF in CapLoader; you can either apply an input filter before loading your PCAPs, or you can apply a display filter after the capture files have been loaded.


Input Filter

The fastest way to filter a large set of PCAP files with CapLoader is to enter an Input Filter before loading the capture files. Having an input filter will speed up the loading time significantly, since CapLoader will not need to analyze packets and flows that don't match the BPF syntax. The downside is that you will need to know beforehand what filter you want to use. In order to apply a changed input filter you need to reload the loaded PCAP files (pressing F5 will do this for you).

CapLoader with input filter “tcp port 443”
Image: CapLoader with input filter “tcp port 443”

Display Filter

CapLoader supports display filters in order to allow filters to be changed on the fly, without having to reload the PCAP files. As the name implies, display filters affect what flows/services/hosts that are displayed in CapLoader. A changed display filter does not require the dataset to be reloaded, but it does require the GUI to update the visible flows. This GUI update will be somewhat slower compared to when setting an input filter.

CapLoader with display filter “host 94.23.23.39”
Image: CapLoader with display filter “host 94.23.23.39”

BPF Syntax

CapLoader's BPF implementation does not support the full BPF syntax. In fact, only the most central primitives are implemented, which are:

host <IP address>Flows to or from the specified IPv4 or IPv6 address
net <CIDR> Flows to or from the specified IP network, uses CIDR notation
port <port>Flows to or from the specified port number
ip6Flows using IPv6 addresses
ipFlows using IPv4 addresses
tcpTCP flows
udpUDP flows
sctpSCTP flows

More complex filter expressions can be built up by using the words and, or, not and parentheses to combine primitives. Here are some examples:

  • host 8.8.8.8 and udp port 53
  • net 199.16.156.0/22 and port 80
  • (port 80 or port 443) and not host 192.168.0.1

For all boolean algebra geeks out there we can confirm that our BPF implementation gives and precedence over or, which means that the last example above would give a different result if the parentheses were removed.


Keeping it Short

Steve McCanne gave a keynote presentation at SharkFest 2011, where he talked about how he created BPF. Steve's work was guided by Van Jacobson, who challenged him to make the BPF syntax human friendly rather than requiring the user to type a clunky filtering syntax. We've adopted this thinking and therefore allow filters like these:

  • 10.1.1.3
    Flows to or from IP address 10.1.1.3. Translates to “ip host 10.1.1.3”

  • 128.3/16
    Flows to or from the 128.3.0.0/16 network. Translates to “ip net 128.3.0.0/16”

  • port 53
    Flows to or from TCP, UDP or SCTP port 53.


Try it for Free!

We've made the BPF implementation available even in the free version of CapLoader. You don't need to register to get the free version; just download, extract and run. The tool is portable, so you won't even have to install it. Visit https://www.netresec.com/?page=CapLoader to grab a copy and start filtering!


UPDATE 2016-05-23

With the release of CapLoader 1.4 it is now possible to apply Display Filters not only to the Flows tab, but also to the Services and Hosts tab.

Posted by Erik Hjelmvik on Monday, 30 November 2015 08:15:00 (UTC/GMT)

Tags: #CapLoader #BPF #PCAP #Berkeley Packet Filter #filter #IP #port #CIDR

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=15B2F44


Split or filter your PCAP files with SplitCap

SplitCap Hatchet

I've released version 1.6 of a tool called SplitCap today. SplitCap is a really fast PCAP file splitter, which can be used to split large pcap files based on for example IP addresses or sessions. You can also use SplitCap in order to filter out traffic to/from specific IP addresses or port numbers from a large pcap file.

The best thing about SplitCap is that it is FAST, much faster than the alternatives.


Split frames in a PCAP file into one file per IP

The -s switch can be used with arguments flow, host, hostpair or nosplit and tells SplitCap how the frames should be separated into the output pcap files.

SplitCap.exe -r huge.pcap -s host
When running the command above SplitCap creates a directory called "huge.pcap" and fills it with lots of pcap files -- one pcap file per identified IP-address (i.e. host). The generated files are named according to the IP address they contain traffic for. Here is how the files from huge.pcap are named:
huge.pcap.Host_128-183-104-74.pcap
huge.pcap.Host_129-33-21-40.pcap
huge.pcap.Host_129-70-4-55.pcap
huge.pcap.Host_129-82-103-72.pcap
huge.pcap.Host_130-212-20-225.pcap
huge.pcap.Host_130-212-255-255.pcap
huge.pcap.Host_131-107-39-4.pcap
huge.pcap.Host_131-107-8-43.pcap
huge.pcap.Host_216-109-126-56.pcap
huge.pcap.Host_64-236-24-4.pcap
huge.pcap.Host_80-67-66-6.pcap
huge.pcap.Host_fe80--202-2dff-fe38-85c0.pcap
huge.pcap.Host_ff02--2.pcap


Split frames based on session

What I believe is a unique feature of SplitCap is the ability to split a pcap file based on session, i.e. the frames from each TCP or UDP session are placed in a separate pcap file. The session split mode is the default one, but you can also use the -s session argument to tell SplitCap to use the session split mode.

SplitCap.exe -r huge.pcap -s session
The generated files get names describing the 5-tuple (proto, src_ip, src_port, dst_ip, dst_port) for which they contain traffic. Here is an example showing how the files can be named:
huge.pcap.TCP_128-183-104-74_80_192-168-21-249_32788.pcap
huge.pcap.TCP_128-183-104-74_80_192-168-21-249_32789.pcap
huge.pcap.TCP_80-67-66-6_80_192-168-22-22_49208.pcap
huge.pcap.TCP_80-67-66-6_80_192-168-22-22_49209.pcap
huge.pcap.UDP_130-212-20-225_1038_239-255-255-250_1900.pcap
huge.pcap.UDP_130-212-20-225_137_130-212-255-255_137.pcap
huge.pcap.UDP_130-212-20-225_138_130-212-255-255_138.pcap
Note that the session parameter tells SplitCap to join frames going both to and from the server into the same pcap file. If you instead wanna separate the two directions into separate pcap files, then you can use the -s flow switch.


Filter PCAP file on IP address

You can filter a pcap file based on address with the -ip switch like this:

SplitCap.exe -r huge.pcap -ip 128.183.104.74 -s nosplit
The -s nosplit argument is used to tell SplitCap not to split the pcap into one file per session. The generated file "huge.pcap.NoSplit.pcap" will only contain frames going to or from the IP address 128.183.104.74. You can also specify multiple IP addresses if you are interested in traffic to/from more than one IP address:
SplitCap.exe -r huge.pcap -ip 128.183.104.74 -ip 80.67.66.6 -s nosplit


Filter PCAP file on port number

What if you are only interested in DNS (UDP+TCP 53) and HTTP (TCP 80) traffic in a PCAP file? Well, then you can specify these port numbers as arguments to SplitCap like this:

SplitCap.exe -r huge.pcap -port 53 -port 80 -s nosplit
This command creates a file named "huge.pcap.NoSplit.pcap" that only contains the frames going to or from ports 53 and 80.


Extract application layer (L7) contents

A pretty cool thing with SplitCap is the ability to extract application (i.e. layer 7) data from a pcap file with the -y L7 argument. I'll use the file "dump.eth0.1059726000.pcap" from Defcon 11 here to demonstrate how to extract layer 7 payload from SMTP (TCP 25) traffic to files (one per session).

SplitCap.exe -r dump.eth0.1059726000.pcap -s session -port 25 -y L7
This command creates 10 files with a ".bin" file extention in the output directory. Each such bin file contains the application layer data for both directions (server->client and vice versa) from an SMTP session.

One of the generated SMTP session files is called "dump.eth0.1059726000.pcap.TCP_64-48-248-30_25_192-168-17-79_33443.bin" and looks something like this (I've redacted some of the contents):

220 smtp102.mail.sc5.yahoo.com ESMTP
[SMTP LOGIN REDACTED]
MAIL FROM: <[REDACTED]@nytimes.com>
250 ok
RCPT TO: <[REDACTED]@[REDACTED].senate.gov>
250 ok
DATA
354 go ahead
Reply-To: <[REDACTED]@nytimes.com>
From: "[REDACTED]" <[REDACTED]@nytimes.com>
To: "[REDACTED]" <[REDACTED]@[REDACTED].senate.gov>
Subject: RE: Just checking in
Date: Fri, 1 Aug 2003 16:28:25 -0400
Message-ID: [REDACTED]
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0700_01C35849.EE94EC60"
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Importance: Normal
In-Reply-To: [REDACTED]

This is a multi-part message in MIME format.

------=_NextPart_000_0700_01C35849.EE94EC60
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: 7bit

i am going to gulfport mississippi for the story. am in las vegas now. will
be in mississippi on sunday.
the guy who is really good on this and helped me actually was [REDACTED]
from uspirg. he put me in contact with a whistle blower.

[REDACTED]
-----Original Message-----
From: [REDACTED] [mailto:[REDACTED]@[REDACTED].senate.gov]
Sent: Friday, August 01, 2003 4:17 PM
To: [REDACTED]@nytimes.com
Subject: Just checking in


I wanted to make sure you are getting everything you need from us and to
check in on the story. Let me know if/when you want to sit down with
[REDACTED] on this. Hope all is well, [REDACTED]


SplitCap is FAST, really FAST!

So how fast is SplitCap at splitting or filtering a pcap file? Let's do a simple benchmark and compare it to the well known tool Tshark. Again, lets use the 189 MB file "dump.eth0.1059726000.pcap" from Defcon 11 for this example.

Filtering the file based on IP with Tshark takes 50 seconds (4 MB/s).

tshark.exe -r dump.eth0.1059726000.pcap -R "ip.addr eq 12.129.71.102" -w defcon11_tshark_filtered.pcap
Performing an equivalent filtering with SplitCap takes 3 seconds (63 MB/s). That is 16 times faster than running Tshark!
SplitCap.exe -r dump.eth0.1059726000.pcap -ip 12.129.71.102 -s nosplit

You can read more about other command line tools from Netresec in the following posts:

SplitCap can be downloaded from here: http://www.netresec.com/?page=SplitCap

Posted by Erik Hjelmvik on Sunday, 08 May 2011 17:41:00 (UTC/GMT)

Tags: #Netresec #pcap #filter

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=1154402

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)