Analyzing Web Browsing Activity

One of the features included in the newly released version 2.0 of NetworkMiner Professional is a new tab called “Browsers”. This tab shows web browsing requests and reponses in a hierarchical tree view, with the identified web browsers as root nodes.

The idea of tracking browser activity this way was suggested to me by Steffen Thorkildsen way back in 2009. I'm therefore happy to finally have this feature implemented in NetworkMiner!

At first glance, the Browser tab looks somewhat like the Hosts tab. One difference is that there can be multiple browsers per host, since each unique HTTP User-Agent is considered a separate browser.

The web pages (URLs) visited by a browser can be analyzed by expanding the node of that browser. The URLs are organized in a hierarchical structure, so that all URLs visited by clicking a link on a web page are placed under the node of that web page. This enables the analyst to see how a user ended up at a particular URL. NetworkMiner primarily uses the HTTP referer header (the misspelling of referrer stems back to RFC1945) to backtrack the pages visited before landing at a particular page.

The browser tree view also shows HTTP redirects, such as “302 Found” and “301 moved permanently”. These redirects can be used in order to see encrypted HTTPS URLs that a user is redirected to, for example when logging in at a website.

Image: Microsoft responding with a “302 Moved Temporarily" redirect

The icons that show up at some web servers are favicon images that have been passively extracted from the analyzed PCAP file.

We hope the Browser tab can be of help in criminal investigations in order to show whether or not a suspect visited a particular website intentionally. This feature can also be used to track the activity of malware that uses HTTP for command-and-control (C2) as well as to analyze redirect chains used for malware downloads.

The PCAP file loaded in the screenshot above originally comes from malware-traffic-analysis.net. Note that our analysis was done by running NetworkMiner in Linux to prevent accidental malware infection. The events shown in NetworkMiner's browser tab matches the description of the redirect chain provided at malware-traffic-analysis.net:

162.144.66.10 port 80 - www.crowdfundingformybusiness.com - Compromised website
185.14.30.37 port 80 - goog1eanalitics.pw - First redirect
178.32.173.105 port 80 - 178.32.173.105 - Second redirect
46.101.59.201 port 80 - osooraudie.co.vu - Nuclear EK

The redirect chain leads to a Nuclear Exploit Kit (SWF file with MD5 695a07cbcac3ca64010e168fe495ff4a, VirusTotal). Later on the Nuclear EK retrieves the file “kernel1.exe”, which seems to be related to the Kelihos botnet.

Posted by Erik Hjelmvik on Thursday, 18 February 2016 13:37:00 (UTC/GMT)

Tags: #NetworkMiner​ #HTTP​ #browser​ #favicon​ #redirect​ #malware​ #NSM​ #PCAP​

NetworkMiner 2.0 Released

I'm proud to announce the release of NetworkMiner 2.0 today! There are several longed-for features that are part of this major release, such as:

• SMB/CIFS parser now supports file extraction from SMB write operations.
• Added parser for SMB2 protocol (read and write).
• Additional IEC-104 commands implemented.
• Added Modbus/TCP parser (as requested by attendees at 4SICS 2014).
• Improved SMTP parser.
• Improved FTP parser.
• Improved DNS parser.
• GUI flickering is heavily reduced when loading PCAP files or doing live sniffing.
• Extraction of web server favicon images (shown in Hosts tab).
• Added "Keyword filter" to several tabs (see more details below).

For those not familiar with NetworkMiner, here's a short summary:

NetworkMiner is a Network Forensic Analysis Tool (NFAT) for Windows (but also works in Linux / Mac OS X / FreeBSD). NetworkMiner can be used as a passive network sniffer/packet capturing tool in order to detect operating systems, sessions, hostnames, open ports etc. without putting any traffic on the network. NetworkMiner can also parse PCAP files for off-line analysis and to regenerate/reassemble transmitted files and certificates from PCAP files.

NetworkMiner has, since the first release in 2007, become a popular tool among incident response teams as well as law enforcement. NetworkMiner is today used by companies and organizations all over the world.

Keyword Filtering

Users of NetworkMiner sometimes run into a needle-in-the-haystack problem when trying to find some specific entry in the Files, Parameters or DNS tab. The technique most analysts have been using so far is to sort the data based a column of interest (by clicking the column header) and then scroll down to the row they are looking for. This method isn't optimal, which is why several users have requested support for keyword filtering. I've personally received several of these requests when teaching my Networks Forensics class.

I'm therefore happy to say that version 2.0 of NetworkMiner has a built-in filtering capability for the data displayed in the following tabs:

• Files
• Parameters
• DNS
• Messages
• Sessions

Image: Parameters tab with filter “user-agent” (case insensitive)

The filter allows rows to be filtered based on one or several keywords. The entered keywords are matched against all text in all columns of the tab. A drop-down menu in the filter bar allows the analyst to chose “Exact Phrase”, “All Words” or “Any Word” as search criteria for the entered keywords.

More Data in the Parameters Tab

We have previously held back on what data we add to the Parameters tab. However, now with the filtering feature in place, we decided to add a lot more information to the Parameters list. Some of the new parameter types available in version 2.0 are:

• HTTP request methods
• HTTP URI's
• HTTP response status codes
• HTTP headers
• SMB Tree Connect AndX Request (attempts to connect to a named file share)
• SMB NT Create AndX Request (mapping of filename to file handle ID)
• SMB2 Connect Requests (attempts to connect to a named file share)
• SMB2 File ID (mapping of filename to file handle ID)
• SMB2 file timestamps (Created, Modified and Accessed).

Image: SMB2 file timestamps shown in Parameters tab

A Warning to Malware Analysts

NetworkMiner has previously appended the “.octet-stream” or “x-msdos-program” extension to all binary files being downloaded over HTTP (since those are the MIME types used for Windows executables). As of version 2.0, however, files named {something}.exe will not be renamed this way. This means that there is now a risk of accidental execution of such files, for example if the user right-clicks an .exe file in NetworkMiner and selects “Open File”.

If you analyze PCAP files that might contain malware, then our recommendation is to perform the analysis on some other operating system than Windows. NetworkMiner runs fine on Linux as well as Mac OS X.

NetworkMiner Professional

On top of the updates provided in the free version of NetworkMiner we have added a few additional useful features to NetworkMiner Professional.

The new features in the Pro version include:

• Advanced OS fingerprinting. Identifies a great range of operating systems and device types (including Apple iOS, Android and many others) based on DHCP traffic.
• Web Browsing Analysis. A new GUI tab called "Browsers" has been added, which shows what URLs each unique browser has visited. More details on this feature will be released shortly in a separate blog post.
• User Settings. Settings in GUI can now be saved to make them persistent between executions.
• Better Export Logs. The CSV format used for exporting data has been improved, we have also added support for XML formatted data export.

Defang Executables

Remember the warning about .exe files downloaded over HTTP no longer get the “.octet-stream” extension? To counter the risk of accidental execution of malware we've added a defang feature to NetworkMiner Professional. When enabled, this feature will rename files like “malware.exe” to “malware.exe_” in order to prevent execution. It is not only .exe files being renamed. At the moment the following file extensions are defanged by appending “_” to the extension:

exe, bat, msi, vb, vbe, vbs, pif, com, scr, jar, cmd, js, jse, ps1 and psc1

The defang feature renames files regardless if they were transmitted over HTTP, FTP, TFTP, SMB, SMB2, SMTP or any other protocol supported by NetworkMiner.

Please note that the defang feature is turned off by default. So if you wanna protect yourself from accidental execution, make sure to enable this feature in the NetworkMiner Settings (available under Tools > Settings).

Upgrade Path for Professional Users

We always provide free minor version upgrades of our software. However, please note that version 2.0 is a major version release, which require a new license to be purchased. Customers with a license for NetworkMiner Professional 1.x can get a 20% discount when purchasing version 2.0 by entering their current license number into the “Additional Information” field of the NetworkMiner Professional order form.

Credits

There are many people who have provided feedback and ideas for improvements that are now part of NetworkMiner 2.0. I would especially like to thank Eric Kollmann (author of Satori) for his impressive ability to find bugs in beta releases. I would also like to thank Ulf Skoglund, Dmitry Shchemelev, @xredumb, Sabin and Andrew Brant for their feedback on improvements in DNS, TCP, HTTP and SMTP parsing. Finally, I wanna give a shout-out to Steffen Thorkildsen for spawning the idea for a browser tracking feature.

Posted by Erik Hjelmvik on Tuesday, 09 February 2016 11:38:00 (UTC/GMT)

Tags: #NetworkMiner​ #SMB​ #SMB2​ #Modbus​ #filter​ #favicon​ #SMTP​ #email​