NETRESEC Network Security Blog - Tag : Extract


NetworkMiner 2.3 Released!

The free and open source network forensics tool NetworkMiner now comes with improved extraction of files and metadata from several protocols as well as a few GUI updates. But the biggest improvements for version 2.3 are in the commercial tool NetworkMiner Professional, which now supports VoIP call audio extraction and playback as well as OSINT lookups of file hashes, IP addresses, domain names and URLs.

I’m happy to announce that NetworkMiner 2.3 now does an even better job than before at extracting files and metadata from several protocols. Improvements have been made in the parsers for the following protocols: HTTP, IEC-104, IPv4, Modbus, SIP, SMB, SMB2, SMTP and SSL/TLS.

We have also added support for the SNMP protocol in NetworkMiner 2.3, so that SNMP community strings can be extracted and displayed on the Parameters and Credentials tabs.

SNMP Community Strings in NetworkMiner's Credential tab

Another change is that timestamps are now displayed using the UTC time zone instead of using the local time zone. We have also fixed a few GUI quirks in order to further improve the usability of the tool.


NetworkMiner Professional

The commercial version of NetworkMiner, i.e. NetworkMiner Professional, comes with several additional improvements which are presented below.

VoIP Call Playback

NetworkMiner Professional has received a new tab called “VoIP”, which enables audio playback of VoIP calls that are using SIP and RTP with G.711 μ-law or A-law encoding (u-Law is primarily used in North America and Japan while A-law is used in Europe and most other parts of the world).

Video: Audio playback and extraction to WAV from the “SIP_CALL_RTP_G711” PCAP file in the Wireshark Sample Captures.

The audio streams from the VoIP calls are also extracted to disk as .WAV files when codecs G.729 or G.711 (u-Law and A-Law) is used. NetworkMiner Professional also attempts to reassemble RTP streams encoded with G.722 to .au files.

OSINT Lookups of IP Addresses, Domains, URLs and File Hashes

Right-clicking a network host in NetworkMiner Professional’s Hosts tab brings up a context menu with options for performing lookups of IP and domain names using external sources. We refer to this method as open-source intelligence (OSINT) because the accessed data resides at publicly available sources.

OSINT IP lookup in NetworkMiner Professional 2.3

Clicking on an OSINT provider brings up a webpage with more detailed information about the selected IP address, such as IBM X-Force, mnemonic Passive DNS, Shodan, UrlQuery or VT. However, if you’re lazy like me, then you’ll probably click the “All above!” option instead, which will bring up all of the sources in separate tabs in your browser.

The full list of OSINT providers available for IP lookups includes APNIC Whois, BFK Passive DNS, Censys, Cymon, DNSTrails, ExoneraTor, Google Public DNS, GreenSnow.co, Hurricane Electric, IBM X-Force, Internet Storm Center, mnemonic Passive DNS, PacketTotal, Shodan, ThreatCrowd, ThreatMiner, UrlQuery and VirusTotal.

The domain name lookup menu contains a similar set of providers: BFK Passive DNS, Cymon, DNSTrails, Google Public DNS, Google Safe Browsing, Hybrid Analysis, IBM X-Force Exchange, mnemonic Passive DNS, MXToolBox, MyWOT, Norton Safe Web, PacketTotal, ThreatCrowd, ThreatMiner, URL Void, UrlQuery, VirusTotal, Website Informer, Webutation and Whoisology.


OSINT URL lookup in NetworkMiner Professional 2.3

Right-clicking a URL in the Browsers tab brings up a similar context menu, which additionally includes the following services for URL lookups: Google Safe Browsing, IBM X-Force, ThreatMiner, URLhaus and UrlQuery.


OSINT file hash lookup in NetworkMiner Professional 2.3

Finally, right-clicking on one of the files that NetworkMiner has extracted from a PCAP file brings up a menu for doing OSINT lookups based on the MD5 or SHA256 hash of the file. The sources used for lookups of hashes include IBM X-Force, PacketTotal, ThreatCrowd, TotalHash, UrlQuery, VirScan.org, Comodo Valkyrie, AlienVault OTX, Hybrid Analysis, ThreatMiner and VirusTotal.

Hybrid Analysis API Integration

Did you know that the malware analysis service Hybrid Analysis provides free API keys to people in the IT security community?

@HybridAnalysis: We are excited to announce that full API keys for file submissions are now available to everyone of the IT security community

This is a great move by the Hybrid Analysis team, and we’re happy to announce that we have leveraged their API in NetworkMiner Professional in order to submit files for analysis directly from within the NetworkMiner GUI. The API integration also enables you to query for an IP on Hybrid Analysis to see which previously submitted samples has communicated with that particular IP address.

Here are the steps required to enable the Hybrid Analysis API integration:


Credits

I would like to thank Chris Sistrunk, Mats Karlsson and Michael Nilsson for suggesting several of the protocol and GUI improvements that have been incorporated into this new release. I’d also like to thank Doug Green and Ahmad Nawawi for discovering and reporting bugs in the IP and SSL parser respectively.


Upgrading to Version 2.3

Users who have purchased a license for NetworkMiner Professional 2.x can download a free update to version 2.3 from our customer portal.

Those who instead prefer to use the free and open source version can grab the latest version of NetworkMiner from the official NetworkMiner page.

FOR GREAT JUSTICE!

Posted by Erik Hjelmvik on Tuesday, 03 April 2018 06:27:00 (UTC/GMT)

Tags: #NetworkMiner #PCAP #OSINT #SMTP #SIP #RTP #VoIP #Network Forensics #extract #Netresec

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1846A54


Analyzing Kelihos SPAM in CapLoader and NetworkMiner

This network forensics video tutorial covers how to analyze SPAM email traffic from the Kelihos botnet. The analyzed PCAP file comes from the Stratosphere IPS project, where Sebastian Garcia and his colleagues execute malware samples in sandboxes. The particular malware sample execution we are looking at this time is from the CTU-Malware-Capture-Botnet-149-2 dataset.

Resources

IOCs
990e5daa285f5c9c6398811edc68a659
e4f7fa6a0846e4649cc41d116c40f97835d3bb7d3d0391d3540482f077aa4493
6c55 5545 0310 4840

Check out our series of network forensic video tutorials for more tips and tricks on how to analyze captured network traffic.

Posted by Erik Hjelmvik on Monday, 19 February 2018 06:37:00 (UTC/GMT)

Tags: #Netresec #PCAP #CapLoader #NetworkMiner #videotutorial #video #tutorial #NetFlow #extract

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=182053B


Hunting AdwindRAT with SSL Heuristics

An increasing number of malware families employ SSL/TLS encryption in order to evade detection by Network Intrusion Detection Systems (NIDS). In this blog post I’m gonna have a look at Adwind, which is a cross-platform Remote Access Trojan (RAT) that has been using SSL to conceal it’s traffic for several years. AdwindRAT typically connects SSL sessions to seemingly random TCP ports on the C2 servers. Hence, a heuristic that could potentially be used to hunt for Adwind RAT malware is to look for SSL traffic going to TCP ports that normally don’t use SSL. However, relying on ONLY that heuristic would generate way too many false positives.

Brad Duncan did an interesting writeup about Adwind RAT back in 2015, where he wrote:

I saw the same certificate information used last week, and it continues this week.
  • commonName = assylias
  • organizationName = assylias.Inc
  • countryName = FR
Currently, this may be the best way to identify Adwind-based post-infection traffic. Look for SSL traffic on a non-standard TCP port using that particular certificate.

Unfortunately, Adwind RAT has evolved to use other CN’s in their new certificates, so looking for “assylias.Inc” will not cut it anymore. However, looking for SSL traffic on non-standard TCP ports still holds on the latest Adwind RAT samples that we’ve analyzed.

The PT Research Attack Detection Team (ADT) sent an email with IDS signatures for detecting AdwindRAT to the Emerging-Sigs mailing list a few days ago, where they wrote:

“We offer one of the ways to detect malicious AdwindRAT software inside the encrypted traffic. Recently, the detection of this malicious program in network traffic is significantly reduced due to encryption. As a result of the research, a stable structure of data fragments was created.”

Not only is it awesome that they were able to detect static patterns in the encrypted data, they also provided 25 PCAP files containing AdwindRAT traffic. I loaded these PCAP files into NetworkMiner Professional in order to have a look at the X.509 certificates. NetworkMiner Professional supports Port-Independent Protocol Identification (PIPI), which means that it will automatically identify the C2 sessions as SSL, regardless of which port that is used. It will also automatically extract the X.509 certificates along with any other parameters that can be extracted from the SSL handshake before the session goes encrypted.

X.509 certificates extracted from AdwindRAT PCAP by NetworkMiner Image: Files extracted from ADT’s PCAP files that mach “Oracle” and “cer”.

In this recent campaign the attackers used X.509 certificates claiming to be from Oracle. The majory of the extracted certificates were exactly 1237 bytes long, so maybe they’re all identical? This is what the first extracted X.509 certificate looks like:

Self-signed Oracle America, Inc. X.509 certificate

The cert claims to be valid for a whopping 100 years!

Self-signed Oracle America, Inc. X.509 certificate

Self-signed, not trusted.

However, after opening a few of the other certificates it's clear that each C2 server is using a unique X.509 certificate. This can be quickly confirmed by opening the parameters tab in NetworkMiner Pro and showing only the Certificate Hash or Subject Key Identifier values.

NetworkMiner Parameters tab showing Certificate Hash values Image: Certificate Hash values found in Adwind RAT’s SSL traffic

I also noted that the CN of the certificates isn’t constant either; these samples use CN’s such as “Oracle America”, “Oracle Tanzania”, “Oracle Arusha Inc.”, “Oracle Leonardo” and “Oracle Heaven”.

The CN field is normally used to specify which domain(s) the certificate is valid for, together with any additinoal Subject Alternative Name field. However, Adwind RAT’s certificates don’t contain any domain name in the CN field and they don’t have an Alternative Name record. This might very well change in future versions of this piece of malware though, but I don’t expect the malware authors to generate a certificate with a CN matching the domain name used by each C2 server. I can therefore use this assumption in order to better hunt for Adwind RAT traffic.

But how do I know what public domain name the C2 server has? One solution is to use passive DNS, i.e. to capture all DNS traffic in order to do passive lookups locally. Another solution is to leverage the fact that the Adwind RAT clients use the Server Name Indication (SNI) when connecting to the C2 servers.

TLS Server Name (aka SNI) and Subject CN values don’t match for AdwindRAT Image: TLS Server Name (aka SNI) and Subject CN values don’t match for AdwindRAT

TLS Server Name (SNI) with matching Subject CN from Google Image: TLS Server Name (SNI) with matching Subject CN from Google.

My conclusion is therefore that Brad’s recommendations from 2015 are still pretty okay, even for the latest wave of Adwind RAT traffic. However, instead of looking for a fix CN string I’d prefer to use the following heuristics to hunt for this type of C2 traffic:

  • SSL traffic to non-standard SSL port
  • Self signed X.509 certificate
  • The SNI domain name in the Client Hello message does not match the CN or Subject Alternative Name of the certificate.

These heuristics will match more than just Adwind RAT traffic though. You’ll find that the exact same heuristics will also help identify other pieces of SSL-enabled malware as well as Tor traffic.

Posted by Erik Hjelmvik on Monday, 04 September 2017 19:01:00 (UTC/GMT)

Tags: #NetworkMiner #SSL #TLS #port #PCAP #PIPI #X.509 #certificate #extract

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1798DC3


NetworkMiner 1.6 Released

We've released version 1.6 of NetworkMiner today!

Confetti in Toronto by Winnie Surya Image credits: Confetti in Toronto by Winnie Surya

The new features in NetworkMiner 1.6 include:

  • Drag-and-Drop
    Reassembled files and images can be opened with external tools by drag-and-dropping items from NetworkMiner's Files or Images tabs onto your favorite editor or viewer.

  • Email extraction
    Improved extraction of emails and attachments sent over SMTP.

  • DNS analysis
    Failed DNS lookups that result in NXDOMAIN and SERVFAIL are displayed in the DNS tab along with the flags in the DNS response.

  • Live sniffing
    Improved live sniffing performance.

  • PCAP-over-IP
    Remote live sniffing enabled by bringing the PCAP-over-IP feature into the free open source version of NetworkMiner.


Identifying Malware DNS lookups

NetworkMiner Professional 1.6 with DNS traffic from the Contagio Kuluoz-Asprox

DNS traffic from the Kuluoz-Asprox botnet (PCAP file available via Contagio)

Note the NXDOMAIN responses and “No” in Alexa top 1 million column in the screenshot above; these domains are probably generated by a domain generation algorithm (DGA).

Live Sniffing with Pcap-over-IP

The PCAP-over-IP functionality enables live sniffing also on non-Windows machines, simply by running tcpdump (or dumpcap) and netcat like this:

# tcpdump -i eth0 -s0 -U -w - | nc localhost 57012
For more information about how to run NetworkMiner in Linux, please read our HowTo install NetworkMiner in Ubuntu Fedora and Arch Linux blog post.

To receive the Pcap-over-IP stream in NetworkMiner, simply press Ctrl+R and select a TCP port.

NetworkMiner Pcap-over-IP

For more information about this feature please see our previous blog post about the PCAP‑over‑IP feature.

NetworkMiner Professional

The professional version of NetworkMiner additionally contains the following improvements of the command line tool NetworkMinerCLI:

  • Enabled reading of PCAP and PcapNG data from standard input (STDIN)
  • Full support for PCAP-over-IP
  • More detailed DNS logging in NetworkMinerCLI's CSV export of DNS responses

The ability to read PCAP data from STDIN with NetworkMinerCLI makes it really simple to do live extraction of emails and email attachments. Here's an example showing how to do live SMTP extraction in Linux:

# tcpdump -i eth0 -s0 -w - port 25 or 587 | mono NetworkMinerCLI.exe -r - -w /var/log/smtp_extraction/

The syntax for extracting emails and attachments in Windows is very similar:

C:\>dumpcap.exe -i 1 -f "port 25 or 587" -w - | NetworkMinerCLI.exe -r -

The TCP ports 25 and 587, which are used in the capture filter above, are the standard port numbers for SMTP. In order to do live extraction of files sent over HTTP, simply use “port 80” as capture filter instead. Likewise, X.509 certificates can also be extracted from HTTPS sessions simply by using “port 443” as capture filter.

Download NetworkMiner 1.6

The most recent release of the free (open source) version of NetworkMiner can be downloaded from SourceForge or our NetworkMiner product page. Paying customers can download an update for NetworkMiner Professional from our customer portal.

Credits

We would like to thank Dan Eriksson (FM CERT) and Lenny Hansson (Danish GovCERT) for submitting bug reports and feature requests.

Posted by Erik Hjelmvik on Monday, 16 June 2014 11:00:00 (UTC/GMT)

Tags: #Netresec #NetworkMiner #Professional #SMTP #Extract #DNS #PCAP-over-IP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1463B56


Extracting Metadata from PcapNG files

Photo by Meighan O'Toole

In our blog post about the Chinese MITM of GitHub we revealed the identity of the anonymous capture file uploader by analyzing metadata available in the PCAP-NG file format. In this blog post we explain what type of meta-data that can be found in PcapNG files, and how to extract it.

The old libpcap format (a.k.a. PCAP) is designed to only store captured network frames. The new PCAP-NG format, however, additionally includes the ability to store meta-data in the capture files. In fact only about 20% of the PcapNG file specification concerns storage of captured frames, the remaining 80% is focused on various types of metadata.

So what types of metadata can a PcapNG file contain? Well, the spec allows features such as MAC address of capturing interface as well as interface speed (in bps) to be stored. But the most commonly used metadata types are:

  • Operating system of the sniffer machine
  • Software used to create the capture file (application name and version)
  • Name of interface from where packets are captured
  • Description of interface from where packets are captured
  • Capture filter used when sniffing packets
  • Cached name resolution entries (mappings from IPv4 or IPv6 addresses to host names)
  • User comments on individual frames (a.k.a. “annotations”),

Extracting Metadata

Some of these metadata types can be displayed in Wireshark by clicking “Statistics > Summary”. Here's the information provided when displaying a Summary in Wireshark on MachineB.pcapng from CloudShark:

Wireshark Summary of MachineB.pcapng

However, what is not shown in this summary view is that MachineB.pcapng additionally contains cached name resolution information for two machines. The easiest way to extract this extra metadata is to upload the capture file to pcapng.com, which will display a list of the metadata that was found in the uploaded file as well as convert the capture file to the old libpcap format (without metadata).

Here's the list of metadata information extracted from MachineB.pcapng by pcapng.com:

Type Value
shb_os Windows Server 2003 Service Pack 2, build 3790
shb_userappl Dumpcap 1.8.4 (SVN Rev 46250 from /trunk-1.8)
if_name \Device\NPF_{D007FF28-3F24-4C1B-8EF5-069A6FB811ED}
if_tresol 0x06
if_os Windows Server 2003 Service Pack 2, build 3790
nres_ip6record fe80::9a03:d8ff:fedb:904b = iPhone-von-b1.local
nres_ip4record 192.168.13.124 = iPhone-von-b1.local
nres_ip6record fe80::8685:6ff:fe23:c95d = iPhone-von-Gurkan.local
nres_ip4record 192.168.13.188 = iPhone-von-Gurkan.local

The metadata apparently contains name resolution info for two iPhones, and we've got their internal IPv4 as well as IPv6 addresses. This would normally imply that the capture file contains frames to or from these IP addresses. However, as shown in the following screenshot from CloudShark, the capture file contains 21 frames but not a single one of them is from or to “iPhone-von-Gurkan” or “iPhone-von-b1”:

CloudShark's view of MachineB.pcapng

Thus, it seems as if the Wireshark pcap-ng host name disclosure bug wasn't fixed in Wireshark 1.8.4 after all.

Megecap Leaks File Paths

It is, however, not only Wireshark that might leak private information in PcapNG metadata. The command line tool Mergecap additionally stores the complete path and filename of the individual capture files that have been combined with the tool.

Type Value
opt_comment File created by merging: File1: /home/erik/captures/customer_X/capture_tap0_p5p1.cap File2: /home/erik/captures/customer_X/capture_tap1_p1p1.cap
shb_userappl mergecap
if_tresol 0x06

Metadata available in PcapNG file created with mergecap

This is most likely not a bug, but rather information leakage by design, so don't expect this to be mitigated in a future Mergecap release.

Conclusions

As we pointed out in our blog post “HowTo handle PcapNG files”, the best way to avoid information leakage via PcapNG files is to stay with the good ol' PCAP format by using the “-P” switch like this:

dumpcap -P -i eth0 -w dump.pcap

Another alternative is to convert your PcapNG files to PCAP format before sharing them.

Posted by Erik Hjelmvik on Thursday, 14 February 2013 19:25:00 (UTC/GMT)

Tags: #PcapNG #pcap-ng #PCAP #Extract

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=132A297


CapLoader Video Tutorial

CapLoader Logo

Below is a short video tutorial showing some of the cool features in CapLoader 1.0.

The functionality showed in the video includes:

  • Loading multiple pcap files into a single flow view
  • Port Independent Protocol Identification (PIPI)
  • Fast extraction of packets related to one or several flows
  • Exporting packets to Wireshark and NetworkMiner
  • Drag-and-dropping packets to Wireshark
  • Selecting a flow based on an IDS alert from Snort
  • Extracting packets from a selected flow to a new pcap file

The video can also be seen on YouTube at the following URI:
http://youtu.be/n1Ir9Hedca4?hd=1

The three pcap files loaded in the video tutorial are from the DFRWS 2009 Challenge.

Enjoy!

Posted by Erik Hjelmvik on Monday, 30 April 2012 14:35:00 (UTC/GMT)

Tags: #CapLoader #Video #Pcap #Wireshark #NetworkMiner #Flow #TCP #Extract #Fast

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=124DA49

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)