NETRESEC Network Security Blog - Tag : regex

rss Google News

CapLoader 1.9.6 Released

CapLoader 1.9.6

CapLoader now detects even more malicious protocols and includes several new features such as JA4 fingerprints, API support for sharing IOCs to ThreatFox and OSINT lookups of malware families on Malpedia. The new CapLoader 1.9.6 release also comes with several improvements of the user interface, for example interactive filtering of flows and services with regular expressions.

Detection of Malware C2 Protocols

Malware authors continually keep coming up with new C2 protocols for defenders to detect. Luckily we don’t need to manually create protocol signatures for CapLoader, we only need a few examples of traffic for a protocol to generate a statistical model that CapLoader can use to detect that protocol. We call this feature Port Independent Protocol Identification (PIPI).

We’ve added support for detecting of the following protocols in this new release of CapLoader:

Malicious protocols detected by CapLoader

Image: Protocols identified in PCAP files with malware traffic from various sandboxes (ANY.RUN, Hybrid-Analysis, Joe Sandbox and Triage)

Our PIPI feature can also detect protocols inside of other protocols, such as Cobalt Strike, DCRat, Emotet, Formbook, Gozi ISFB, GzipLoader and Socks5Systemz which all run on top of HTTP. It is sometimes even possible to identify malicious protocols that use TLS encryption, such as AsyncRAT, Cobalt Strike, Emotet, IcedID or Remcos. However, detection of malicious TLS encrypted protocols is a difficult challenge and might be subject to false positives.

Sharing IOCs to ThreatFox

ThreatFox is a free online service for sharing indicators of compromise (IOCs) from malware. ThreatFox can be queried for a particular malware family, such as RedLine Stealer, and it’ll return a list of URLs, domain names and IP:port pairs used for C2 communication or payload delivery for that malware. You can also query for a domain or IP address to see if it’s a known C2 address of any malware or botnet.

CapLoader has supported OSINT lookup of IP addresses and domains on ThreatFox since the release of version 1.9, but with this release we also add the ability to contribute by sharing IOCs with the infosec community. All you need to do is to enter your ThreatFox API-key in CapLoader’s settings, then right-click a flow, service or alert and select “Submit to ThreatFox”.

Submitting Loda IOC to ThreatFox

Image: Submission of microsoft.net.linkpc[.]net to ThreatFox

If the right-clicked item is an alert for a “Malicious protocol” then CapLoader will automatically populate the Mapledia Name field, as shown in the screenshot (win.loda).

TLS Client Fingerprinting with JA4

John Althouse announced the new JA4+ fingerprint methods a couple of months ago on the FoxIO blog. In short JA4+ is a suite of methods designed to fingerprint implementations of a specific set of protocols, including TLS, HTTP and SSH. As you’ve probably guessed JA4+ is a successor to the JA3 and JA3S hashes that we’ve learned to love (we added JA3 fingerprinting to NetworkMiner in 2019).

Most of the fingerprinting methods in the JA4+ suite are patent pending except for the TLS client fingerprinting method “JA4”, which FoxIO does not have patent claims and is not planning to pursue patent coverage for. We have therefore built a JA4 fingerprinting engine that we’ve included in this CapLoader release. Future releases of NetworkMiner will hopefully also include our JA4 fingerprinting engine.

JA3 and JA4 fingerprints of Remcos traffic. a85be79f7b569f1df5e6087b69deb493 t13i010400_0f2cb44170f4_5c4c70b73fa0 t13i010400_0f2cb44170f4_1b583af8cc09

Image: JA3 and JA4 hashes of Remcos C2 traffic

JA4 is similar to JA3 in many ways, but one essential difference is that JA4 fingerprints are something of a fuzzy hash of the client’s handshake rather than a MD5 hash of the raw fingerprint. JA3’s use of MD5 hashing has received criticism, for example in academic literature, partly due to the inability to see if two JA3 hashes have similar TLS handshakes.

JA4 hash explained. Breakdown of Remcos JA4 hash t13i010400_0f2cb44170f4_5c4c70b73fa0

JA4 does use hashes, but instead of just being one big hash it breaks the fingerprint into three separate sections; where the first section is used in its raw (non-hashed) format and the other two sections are hashed separately. Thus, an update of a TLS implementation, which only adds one additional cipher, will increment the cipher counter in the first section of the JA4 fingerprint by one and the ciphers hash (second section) will get a new value. The hash in the last section will remain intact.

In the previous CapLoader screenshot with Remcos C2 traffic we see TLS handshakes that have the same JA3 hash (a85be79f7b569f1df5e6087b69deb493) but the JA4 fingerprints have different values (t13i010400_0f2cb44170f4_5c4c70b73fa0 and t13i010400_0f2cb44170f4_1b583af8cc09). The reason why the last JA4 section is different even though the JA3 hash is the same is because some of these TLS handshakes present a different set of signature algorithms, which is a parameter that isn't being used in JA3.

Alerts Tab

CapLoader’s Alerts tab now includes more alert types than before and each alert has a severity rating graded as follows:

  • High = 4
  • Medium = 3
  • Low = 2
  • Info = 1

A typical high-severity alert is when a known malicious protocol is detected, while an “Info” type alert can provide a heads up about traffic from things like coin mining or legitimate remote admin tools. As you can see in the screenshot below the alerts are sorted based on severity to make it easier to prioritize them.

Alerts in CapLoader for 2023-10-16-IcedID-infection.pcap

Image: CapLoader alerts for 2023-10-16-IcedID-infection.pcap

Here’s a breakdown of the alerts shown in the CapLoader screenshot above:

All these alerts are indicators of an IcedID infection, including the 5 minute C2 connection interval which I have mentioned before.

Other User Interface Improvements

CapLoader’s “Column Criteria” row filter could previously only be used to filter on columns with a specific value, such as “Protocol = TLS”. This new release of CapLoader additionally allows users to do substring matching with the “contains” keyword and regular expression (regex) matching with the “matching” keyword. In the screenshot below the Column Criteria “Hostname matches \.local$” is used to only show hosts that have a hostname ending with “.local”.

RegEx matching of .local hostnames

We’ve also added an often asked for feature to CapLoader, namely the ability to switch between different flows in the Transcript window.

CapLoader Transcript. Change this number to show next flow

The flows you can switch between depends on how the transcript window was opened. A flow transcript opened from the Flows tab will allow switching between the flows that were visible in the list from where the transcript was opened. A transcript opened from any of the other tabs (Services, Hosts or Alerts), on the other hand, allows switching between the different flows for the particular service, host or alert that was opened.

Credits

I would like to thank Nic Cerny, Trent Healy and Fredrik Ginsberg for their input on various improvements that have been implemented in CapLoader 1.9.6.

Updating to the Latest Release

Users who have already purchased a license for CapLoader can download a free update to version 1.9.6 from our customer portal or by clicking “Check for Updates” in CapLoader’s Help menu.

Posted by Erik Hjelmvik on Wednesday, 15 November 2023 12:08:00 (UTC/GMT)

Tags: #CapLoader#ThreatFox#JA3#IcedID#GzipLoader#regex

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=23B6bcd


CapLoader 1.8 Released

CapLoader 1.8

We are happy to announce the release of CapLoader 1.8 today!

CapLoader is primarily used to filter, slice and dice large PCAP datasets into smaller ones. This new version contains several new features that improves this filtering functionality even further. To start with, the “Keyword Filter” can now be used to filter the rows in the Flows, Services or Hosts tabs using regular expressions. This enables the use of matching expressions like this:

  • amazon|akamai|cdn
    Show only rows containing any of the strings “amazon” “akamai” or “cdn”.
  • microsoft\.com\b|windowsupdate\.com\b
    Show only servers with domain names ending in “microsoft.com” or “windowsupdate.com”.
  • ^SMB2?$
    Show only SMB and SMB2 flows.
  • \d{1,3}\.\d{1,3}\.\d{1,3}\.255$
    Show only IPv4 address ending with “.255”.

For a reference on the full regular expression syntax available in CapLoader, please see Microsoft’s regex “Quick Reference”.

One popular workflow supported by CapLoader is to divide all flows (or hosts) into two separate datasets, for example one “normal” and one “malicious” set. The user can move rows between these two sets, where only one set is visible while the rows in the other set are hidden. To switch which dataset that is visible versus hidden the user needs to click the [Invert Hiding] button (or use the [Ctrl]+[Tab] key combination). With this new release we’ve also made the “Invert Hiding” functionality available by clicking the purple bar, which shows the number of rows present in the currently viewed set.

CapLoader Invert Hiding GIF

Readers with a keen eye might also notice that the purple bar charts are now also accompanied by a number, indicating how many rows that are visible after each filter is applied. The available filters are: Set Selection, BPF and Keyword Filter.

NetFlow + DNS = Great Success!

CapLoader’s main view presents the contents of the loaded PCAP files as a list of netflow records. Since the full PCAP is available, CapLoader also parses the DNS packets in the capture files in order to enrich the netflow view with hostnames. Recently PaC shared a great idea with us, why not show how many failed DNS lookups each client does? This would enable generic detection of DGA botnets without using blacklists. I’m happy to announce that this great idea made it directly into this new release! The rightmost column in CapLoader’s hosts tab, called “DNS_Fails”, shows how many percent of a client’s DNS requests that have resulted in an NXRESPONSE or SRVFAIL response.

CapLoader 1.8

Two packet capture files are loaded into CapLoader in the screenshot above; one PCAP file from a PC infected with the Shifu malware and one PCAP file with “normal traffic” (thanks @StratosphereIPS for sharing these capture files). As you can see, one of the clients (10.0.2.107) has a really high DNS failure ratio (99.81%). Unsurprisingly, this is also the host that was infected with the Shifu, which uses a domain generation algorithm (DGA) to locate its C2 servers.

Apart from parsing A and CNAME records from DNS responses CapLoader now also parses AAAA DNS records (IPv6 addresses). This enables CapLoader to map public domain names to hosts with IPv6 addresses.

Additional Updates

The new CapLoader release also comes with several other new features and updates, such as:

  • Added urlscan.io service for domain and IP lookups (right-click a flow or host to bring up the lookup menu).
  • Flow ID coloring based on 5-tuple, and clearer colors in timeline Gantt chart.
  • Extended default flow-timeout from 10 minutes to 2 hours for TCP flows.
  • Changed flow-timout for non-TCP flows to 60 seconds.
  • Upgraded to .NET Framework 4.7.2.

Updating to the Latest Release

Users who have previously purchased a license for CapLoader can download a free update to version 1.8 from our customer portal. All others can download a free 30 day trial from the CapLoader product page (no registration required).

Credits

We’d like to thank Mikael Harmark, Mandy van Oosterhout and Ulf Holmström for reporting bugs that have been fixed in this release. We’d also like to thank PaC for the DNS failure rate feature request mentioned in this blog post.

Posted by Erik Hjelmvik on Tuesday, 28 May 2019 10:45:00 (UTC/GMT)

Tags: #CapLoader#NetFlow#regex#DNS#DGA#Stratosphere

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1950482


Detecting the Pony Trojan with RegEx using CapLoader

This short video demonstrates how you can search through PCAP files with regular expressions (regex) using CapLoader and how this can be leveraged in order to improve IDS signatures.

The EmergingThreats snort/suricata rule mentioned in the video is SID 2014411 “ET TROJAN Fareit/Pony Downloader Checkin 2”.

The header accept-encoding header with quality factor 0 used by the Pony malware is:
Accept-Encoding: identity, *;q=0

And here is the regular expression used to search for that exact header: \r\nAccept-Encoding: identity, \*;q=0\r\n

After recording the video I noticed that the leaked source code for Pony 2.0 actually contains this accept-encoding header as a hard-coded string. Have a look in the redirect.php file, where they set curl’s CURLOPT_HTTPHEADER to this specific string.

Pony using curl to set: Accept-Encoding: identity, *;q=0

Wanna learn more about the intended use of quality factors in HTTP accept headers? Then have a look at section 14.1 of RFC 2616section 5.3.4 of RFC 7231, which defines how to use qvalues (i.e. quality factors) in the Accept-Encoding header.

Finally, I'd like to thank Brad Duncan for running the malware-traffic-analysis.net website, your PCAP files often come in handy!

Update 2018-07-05

I submitted a snort/suricata signature to the Emerging-Sigs mailinglist after publishing this blog post, which resulted in the Emerging Threats signature 2014411 being updated on that same day to include:

content:"|0d 0a|Accept-Encoding|3a 20|identity,|20 2a 3b|q=0|0d 0a|"; http_header;

Thank you @EmergingThreats for the fast turnaround!

Posted by Erik Hjelmvik on Wednesday, 04 July 2018 07:39:00 (UTC/GMT)

Tags: #video#regex#malware#IDS#curl#malware-traffic-analysis.net#videotutorial

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=187e291


CapLoader 1.7 Released

We are happy to announce the release of CapLoader 1.7! CapLoader 1.7 logo

Here’s an overview of what’s new in this release:

  • Regular expression searching
  • Lookup of IP addresses using online services
  • Lookup of domain names using online services
  • Improved protocol fingerprinting speed and precision
  • Support for GRE, IGMP and ICMPv6 flows
  • More precise period estimation of “periodic services


Regular Expressions Search

CapLoader’s “Find Keyword” window has been extended with an option to search flows using regular expressions (regex) as an alternative to searching for strings or byte sequences. With help of the powerful regex syntax built into .NET this new search option enables very flexible searching.

Searching for Alfa Ransomware flows in CapLoader using regex
Image: Searching for Alfa Ransomware flows using regex

The Find Keyword window can be opened by clicking Edit > Find Keyword, or by pressing Ctrl+F after having loaded a PCAP file.

You can learn more about regex searching with CapLoader in our short video called "Detecting the Pony Trojan with RegEx using CapLoader".


OSINT Lookups of IP Addresses and Domains

We added support for querying online services for IPs and domain names to the latest release of NetworkMiner, now it’s time to add this very handy feature to CapLoader as well.

Right-clicking a Flow, Service or Host in CapLoader brings up a context menu with links to various online resources that might have more details regarding the clicked IP address or domain name.

CapLoader OSINT lookups

The services available for IP address OSINT lookup include:
APNIC Whois, Censys, Cymon, ExoneraTor, Google Public DNS, GreenSnow.co, Hurricane Electric, IBM X-Force, Internet Storm Center, mnemonic Passive DNS, PacketTotal, SecurityTrails, Shodan, ThreatCrowd, ThreatMiner, UrlQuery and VirusTotal.

The domain name lookup menu contains a similar set of providers:
Cymon, Google Public DNS, Google Safe Browsing, Hybrid Analysis, IBM X-Force Exchange, mnemonic Passive DNS, MXToolBox, MyWOT, Norton Safe Web, PacketTotal, SecurityTrails, ThreatCrowd, ThreatMiner, URL Void, UrlQuery, VirusTotal, Website Informer, Webutation and Whoisology.


Protocol Identification

The dynamic protocol detection (or Port Independent Protocol Detection, aka “PIPI”) in CapLoader has been improved to support even more protocols than before. We have also fine-tuned the protocol identification algorithm to be both faster and more accurate.

CapLoader 1.7 identifying SSL on non-standard port
Image: Traffic to TCP 8777 identified as SSL (PCAP file from Stratosphere IPS)


Updating to the Latest Release

Users who have previously purchased a license for CapLoader can download a free update to version 1.7 from our customer portal. All others can download a free 30 day trial from the CapLoader product page (no registration required).


Credits

We’d like to thank Michael Nilsson for suggesting the IP and domain name lookup feature and Tohar Braun for suggesting RegEx search support. We’d also like to thank Ralf Alvarsson and Jarmo Lahtiranta for reporting bugs that have been resolved in this release.

Posted by Erik Hjelmvik on Tuesday, 03 July 2018 11:37:00 (UTC/GMT)

Tags: #CapLoader#regex#keyword#pcap#GRE#OSINT#IP

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1878305

X / twitter

NETRESEC on X / Twitter: @netresec

Mastodon

NETRESEC on Mastodon: @netresec@infosec.exchange