Optimizing IOC Retention Time

Are you importing indicators of compromise (IOC) in the form of domain names and IP addresses into your SIEM, NDR or IDS? If so, have you considered for how long you should keep looking for those IOCs?

An IoT botnet study from 2022 found that 90% of C2 servers had a lifetime of less than 5 days and 93% had a lifetime shorter than 14 days. Additionally, a recent writeup from Censys concludes that the median lifespan of Cobalt Strike C2 servers is 5 days. Both these studies indicate that IP and domain name indicators are short lived, yet many organizations cling on to old IOCs for much longer than so. Monitoring for too many old indicators not only costs money, it can even inhibit detection of real intrusions.

Pyramid of Pain

David J. Bianco's pyramid of pain illustrates how much pain is caused to an adversary if their malicious actions are hindered by defenders blocking various indicators.

Indicators at the bottom of the pyramid are trivial for an adversary to replace, which is why IOCs like hashes and IP addresses tend to be very short lived. The indicator types close to the pyramid’s base are also the ones that typically are shared in threat intel feeds.

It would be nice to detect adversaries with indicators higher up in David’s pyramid of pain, but it is often difficult to craft reliable detection mechanisms for such indicators. The simple indicators at the bottom, like domains, IPs and hashes, are on the other hand very exact and practical for us defenders to work with. So in this sense the pyramid of pain applies to defenders as well, where indicators at the bottom are trivial to use and the complexity increases as we go further up. In this sense the Pyramid of Pain can also be viewed as the “Pyramid of Detection Complexity”.

Chasing Ghosts

Many IOCs are “dead” even before you get them. This IOC delay is particularly noticeable when looking at writeups of malware and botnets published by security researchers, in which the IPs and domain names shared in their IOC lists often haven’t been used for several weeks by the time the report gets published. It’s understandable that it takes time to reverse engineer a new piece of malware, compose a blog post and to get the writeup approved for publishing. Nevertheless, those old IOCs often get picked up, redistributed, and used as if they were fresh indicators. Recorded Future observed a 33-day average lead time between when their scans found a C2 server and when it is reported in other sources. Thirty-three days! Only a very small fraction of those C2 servers can be expected to still be active by the time those IOCs are shared by various threat intel providers. All the other C2 servers are dead indicators, or “Ghost IOCs” as I like to call them. Attempting to detect such Ghost IOCs in live network traffic is a waste of resources.

IOC Costs

Most network intrusion detection systems (IDS) only support a certain number of active rules or signatures, which effectively limits the number of IOCs you can look for. Other products charge the user based on the number of monitored IOCs. As an example, Tostes et al.’s paper covering the shelf life of an IOC says that Azure Sentinel Threat Intelligence charges $2.46 per ingested GB. This type of direct cost can then be compared to the cost of not alerting on a known malicious IOC that has timed out. Simplified comparisons like this typically result in poor and misleading guidance to keep using IOCs for much longer than necessary, often hundreds of days or even perpetually.

One important factor that is missing in such simplified reasoning is the cost for false positive alerts, which increases significantly when monitoring for ghost IOCs. An IP address that is being used as a botnet C2 server one week might be running a totally legitimate service another week. The risk for false positives is smaller for domain names compared to IP addresses, but hacked websites is one example where domain names often cause false positive alerts as well. One common cause for such domain IOC false positives is the frequent misuse of hacked websites for malware distribution. Such illicit use is often detected and rectified within a couple of days, in particular when this happens to a website belonging to a medium to large sized company or organization. Alerting on DNS based IOCs for a long time after last confirmed sighting therefore also increases the risk for false positives.

The cost of false positives is difficult to estimate, as it involves the time analysts spend in vain trying to verify if a particular alert was a false positive or not. Too many false positives can also cause alert fatigue or alarm burnout, much in the same way as in Aesop’s ancient fable about the boy who cried wolf.

This implies that monitoring for too many old IOCs actually increases the risk of actual sightings of malicious indicators being missed or overlooked due to alert fatigue. This line of thinking might seem contradictory and is often overlooked in research related to the shelf life of indicators. I would personally prefer to use an IOC scoring model that aims to reduce the number of false positives rather than using one that attempts to minimize the direct cost of IOC monitoring.

Pruning old IOCs

RFC 9424 states that “IOCs should be removed from detection at the end of their life to reduce the likelihood of false positives”. But how do we know if an IOC has reached end of life?

I’ve previously mentioned that researchers have found that most C2 servers are used for 5 days or less. The optimal number of days to monitor for an IOC depends on several factors, but in general we’re talking about a couple of weeks since last sighting for an IP address and maybe a little longer for domain names. The obvious exception here is DGA and RDGA domains, which typically have a lifetime of just 24 hours.

The research paper Taxonomy driven indicator scoring in MISP threat intelligence platforms introduces an interesting concept, where a score is calculated for each IOC based on factors like confidence, age and decay rate. The IOC can be considered dead or expired when the score reaches zero or if it goes below a specified threshold. The following graph from the MISP paper shows an example of how the score for an IP address IOC could decay over time.

Image: Figure 8 from Taxonomy driven indicator scoring in MISP threat intelligence platforms

Recommendations

• When publishing IOCs, make sure to also include a date for when it was last confirmed to be active, aka “last seen”.
• Ask your threat intel vendor if they can provide a last seen date for each of their indicators, or if they have some other way to determine the freshness of their indicators.
• Ensure that the IOCs you monitor for are pruned based on age or freshness to reduce the risk (and cost) for false positives.
• Prioritize long lived IOCs over short lived ones, for example by striving to use indicators higher up in the pyramid of pain, but only when this can be done without increasing the false positive rate.

Posted by Erik Hjelmvik on Thursday, 06 November 2025 12:05:00 (UTC/GMT)

Tags: #IOC​ #IDS​ #ASCII-art​

Sniffing Decrypted TLS Traffic with Security Onion

Wouldn't it be awesome to have a NIDS like Snort, Suricata or Zeek inspect HTTP requests leaving your network inside TLS encrypted HTTPS traffic? Yeah, we think so too! We have therefore created this guide on how to configure Security Onion to sniff decrypted TLS traffic with help of PolarProxy.

PolarProxy is a forward TLS proxy that decrypts incoming TLS traffic from clients, re-encrypts it and forwards it to the server. One of the key features in PolarProxy is the ability to export the proxied traffic in decrypted form using the PCAP format (a.k.a. libpcap/tcpdump format). This makes it possible to read the decrypted traffic with external tools, without having to perform the decryption again. It also enables packet analysis using tools that don't have built-in TLS decryption support.

This guide outlines how to configure PolarProxy to intercept HTTPS traffic and send the decrypted HTTP traffic to an internal network interface, where it can be sniffed by an IDS.

STEP 1 ☆ Install Ubuntu

Download and install the latest SecurityOnion ISO image, but don't run the "Setup" just yet.

STEP 2 ☆ Add a Dummy Network Interface

Add a dummy network interface called "decrypted", to which decrypted packets will be sent.

STEP 3 ☆ Install Updates

Install updates in Security Onion by running "sudo soup".

STEP 4 ☆ Run the Security Onion Setup

Run the Security Onion setup utility by double-clicking the "Setup" desktop shortcut or executing "sudo sosetup" from a terminal. Follow the setup steps in the Production Deployment documentation and select "decrypted" as your sniffing interface.

Reboot and run Setup again to continue with the second phase of Security Onion's setup. Again, select "decrypted" as the interface to be monitored.

STEP 5 ☆ Install PolarProxy Service

Download and install PolarProxy:

Edit /etc/systemd/system/PolarProxy.service and add "--pcapoverip 57012" at the end of the ExecStart command.

Start the PolarProxy systemd service:

STEP 6 ☆ Install Tcpreplay Service

The decrypted traffic can now be accessed via PolarProxy's PCAP-over-IP service on TCP 57012. We can leverage tcpreplay and netcat to replay these packets to our dummy network interface in order to have them picked up by Security Onion.

Start the tcpreplay systemd service:

STEP 7 ☆ Add firewall rules

Security Onion only accepts incoming connections on TCP 22 by default, we also need to allow connections to TCP port 10443 (proxy port), and 10080 (root CA certificate download web server). Add allow rules for these services to the Security Onion machine's firewall:

Verify that the proxy is working by running this curl command on a PC connected to the same network as the Security Onion machine:

Add the following lines at the top of /etc/ufw/before.rules (before the *filter section) to redirect incoming packets on TCP 443 to PolarProxy on port 10443.

Note: Replace "enp0s3" with the Security Onion interface to which clients will connect.

After saving before.rules, reload ufw to activate the port redirection:

Verify that you can reach the proxy on TCP 443 with this command:

STEP 8 ☆ Redirect HTTPS traffic to PolarProxy

It's now time to configure a client to run its HTTPS traffic through PolarProxy. Download and install the PolarProxy X.509 root CA certificate from PolarProxy's web service on TCP port 10080:

http://[SecurityOnionIP]:10080/polarproxy.cer

Install the certificate in the operating system and browser, as instructed in the PolarProxy documentation.

You also need to forward packets from the client machine to the Security Onion machine running PolarProxy. This can be done either by configuring a local NAT rule on each monitored client (STEP 8.a) or by configuring the default gateway's firewall to forward HTTPS traffic from all clients to the proxy (STEP 8.b).

STEP 8.a ☆ Local NAT

Use this firewall rule on a Linux client to configure it to forward outgoing HTTPS traffic to the Security Onion machine:

STEP 8.b ☆ Global NAT

If the client isn't running Linux, or if you wanna forward HTTPS traffic from a whole network to the proxy, then apply the following iptables rules to the firewall in front of the client network. See "Routing Option #2" in the PolarProxy documentation for more details.

1. Add a forward rule on the gateway to allow forwarding traffic to our PolarProxy server:
   sudo iptables -A FORWARD -i eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -m state --state NEW -j ACCEPT
2. Add a DNAT rule to forward 443 traffic to PolarProxy on port 10443:
   sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j DNAT --to [SecurityOnionIP]:10443
3. If the reverse traffic from PolarProxy to the client doesn't pass the firewall (i.e. they are on the same LAN), then we must add this hide-nat rule to fool PolarProxy that we are coming from the firewall:
   sudo iptables -t nat -A POSTROUTING -o eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -j MASQUERADE

STEP 9 ☆ Inspect traffic in SecurityOnion

Wait for the Elastic stack to initialize, so that the intercepted network traffic becomes available through the Kibana GUI. You can check the status of the elastic initialization with "sudo so-elastic-status".

You should now be able to inspect decrypted traffic in Security Onion using Kibana, Squert, Sguil etc., just as if it was unencrypted HTTP.

Image: Kibana showing HTTP traffic info from decrypted HTTPS sessions

Image: MIME types in Kibana

Image: NIDS alerts from payload in decrypted traffic shown in Kibana

Image: Snort alerts from decrypted traffic shown in Squert

Security Considerations and Hardening

Security Onion nodes are normally configured to only allow access by SOC/CERT/CSIRT analysts, but the setup described in this blog post requires that "normal" users on the client network can access the PolarProxy service running on the Security Onion node. We therefore recommend installing PolarProxy on a dedicated Security Onion Forward Node, which is configured to only monitor traffic from the proxy.

We also recommend segmenting the client network from the analyst network, for example by using separate network interfaces on the Security Onion machine or putting it in a DMZ. Only the PolarProxy service (TCP 10080 and 10443) should be accessable from the client network.

PolarProxy could be used to pivot from the client network into the analyst network or to access the Apache webserver running on the Security Onion node. For example, the following curl command can be used to access the local Apache server running on the Security Onion machine via PolarProxy:

Hardening Steps:

• Configure the Security Onion node as a Forward Node
• Segment client network from analyst network
• Add firewall rules to prevent PolarProxy from accessing services on the local machine and analyst network

For additional info on hardening, please see the recommendations provided by Wes Lambert on the Security-Onion mailing list.

Posted by Erik Hjelmvik on Monday, 20 January 2020 09:40:00 (UTC/GMT)

Tags: #SecurityOnion​ #Security Onion​ #PCAP​ #Bro​ #Zeek​ #PolarProxy​ #Snort​ #Suricata​ #TLS​ #SSL​ #HTTPS​ #tcpreplay​ #PCAP-over-IP​ #IDS​ #NIDS​ #netcat​ #curl​ #UFW​ #ASCII-art​

PolarProxy Released

I’m very proud to announce the release of PolarProxy today! PolarProxy is a transparent TLS proxy that decrypts and re-encrypts TLS traffic while also generating a PCAP file containing the decrypted traffic.

PolarProxy enables you to do lots of things that have previously been impossible, or at least very complex, such as:

• Analyzing HTTP/2 traffic without an SSLKEYLOGFILE
• Viewing decrypted HTTPS traffic in real-time using Wireshark
  PolarProxy -p 10443,80,443 -w - | wireshark -i - -k
• Replaying decrypted traffic to an internal or external interface using tcpreplay
  PolarProxy -p 10443,80,443 -w - | tcpreplay -i eth1 -
• Forwarding of decrypted traffic to a NIDS (see tcpreplay command above)
• Extracting DNS queries and replies from DNS-over-TLS (DoT) or DNS-over-HTTPS (DoH) traffic
  PolarProxy -p 853,53 -p 443,80 -w dns.pcap
• Extracting email traffic from SMTPS, POP3S or IMAPS
  PolarProxy -p 465,25 -p 995,110 -p 993,143 -w emails.pcap

Here is an example PCAP file generated by PolarProxy:
https://media.netresec.com/pcap/polarproxy-demo.pcap

This capture files contains HTTP, WebSocket and HTTP/2 packets to Mozilla, Google and Twitter that would otherwise have been encrypted with TLS.

Now, head over to our PolarProxy page and try it for yourself (it’s free)!

Posted by Erik Hjelmvik on Friday, 21 June 2019 06:00:00 (UTC/GMT)

Tags: #PolarProxy​ #PCAP​ #NIDS​ #IDS​ #http2​ #HTTP/2​ #SSLKEYLOGFILE​ #Wireshark​ #tcpreplay​ #DoH​ #SMTPS​ #IMAPS​ #TLS​ #SSL​

Detecting the Pony Trojan with RegEx using CapLoader

This short video demonstrates how you can search through PCAP files with regular expressions (regex) using CapLoader and how this can be leveraged in order to improve IDS signatures.

The EmergingThreats snort/suricata rule mentioned in the video is SID 2014411 “ET TROJAN Fareit/Pony Downloader Checkin 2”.

The header accept-encoding header with quality factor 0 used by the Pony malware is:
Accept-Encoding: identity, *;q=0

And here is the regular expression used to search for that exact header: \r\nAccept-Encoding: identity, \*;q=0\r\n

After recording the video I noticed that the leaked source code for Pony 2.0 actually contains this accept-encoding header as a hard-coded string. Have a look in the redirect.php file, where they set curl’s CURLOPT_HTTPHEADER to this specific string.

Wanna learn more about the intended use of quality factors in HTTP accept headers? Then have a look at section 14.1 of RFC 2616section 5.3.4 of RFC 7231, which defines how to use qvalues (i.e. quality factors) in the Accept-Encoding header.

Finally, I'd like to thank Brad Duncan for running the malware-traffic-analysis.net website, your PCAP files often come in handy!

Update 2018-07-05

I submitted a snort/suricata signature to the Emerging-Sigs mailinglist after publishing this blog post, which resulted in the Emerging Threats signature 2014411 being updated on that same day to include:

Thank you @EmergingThreats for the fast turnaround!

Posted by Erik Hjelmvik on Wednesday, 04 July 2018 07:39:00 (UTC/GMT)

Tags: #video​ #regex​ #malware​ #IDS​ #curl​ #malware-traffic-analysis.net​ #videotutorial​

Examining an x509 Covert Channel

Jason Reaves gave a talk titled “Malware C2 over x509 certificate exchange” at BSides Springfield 2017, where he demonstrated that the SSL handshake can be abused by malware as a covert command-and-control (C2) channel.

He got the idea while analyzing the Vawtrak malware after discovering that it read multiple fields in the X.509 certificate provided by the server before proceeding. Jason initially thought these fields were used as a C2 channel, but then realized that Vawtrak performed a variant of certificate pinning in order to discover SSL man-in-the-middle attempts.

Nevertheless, Jason decided to actually implement a proof-of-concept (PoC) that uses the X.509 certificate as a C2 channel. Jason’s code is now available on GitHub along with a PCAP file demonstrating this covert C2 channel. Of course I couldn’t resist having a little look at this PCAP file in NetworkMiner.

The first thing I noticed was that the proof-of-concept PCAP ran the SSL session on TCP 4433, which prevented NetworkMiner from parsing the traffic as SSL. However, I was able to parse the SSL traffic with NetworkMiner Professional just fine thanks to the port-independent-protocol-identification feature (a.k.a Dynamic Port Detection), which made the Pro-version parse TCP 4433 as SSL/TLS.

A “normal” x509 certificate size is usually around 1kB, so certificates that are 11kB should be considered as anomalies. Also, opening one of these .cer files reveals an extremely large value in the Subject Key Identifier field.

Not only is this field very large, it also starts with the familiar “4D 5A” MZ header sequence.

NetworkMiner additionally parses details from the certificates that it extracts from PCAP files, so the Subject Key Identifier field is actually accessible from within NetworkMiner, as shown in the screenshot below.

You can also see that NetworkMiner validates the certificate using the local trusted root certificates. Not surprisingly, this certificates is not trusted (certificate valid = FALSE). It would be most unlikely that anyone would manage to include arbitrary data like this in a signed certificate.

Extracting the MZ Binary from the Covert X.509 Channel

Even though NetworkMiner excels at pulling out files from PCAPs, this is definitively an occasion where manual handling is required. Jason’s PoC implementation actually uses a whopping 79 individual certificates in order to transfer this Mimikatz binary, which is 785 kB.

Here’s a tshark oneliner you can use to extract the Mimikatz binary from Jason's example PCAP file.

tshark -r mimikatz_sent.pcap -Y 'ssl.handshake.certificate_length gt 2000' -T fields -e x509ce.SubjectKeyIdentifier -d tcp.port==4433,ssl | tr -d ':\n' | xxd -r -p > mimikatz.exe

Detecting x509 Anomalies

Even though covert channels using x509 certificates isn’t a “thing” (yet?) it’s still a good idea to think about how this type of covert signaling can be detected. Just looking for large Subject Key Identifier fields is probably too specific, since there are other fields and extensions in X.509 that could also be used to transmit data. A better approach would be to alert on certificates larger than, let’s say, 3kB. Multiple certificates can also be chained together in a single TLS handshake certificate record, so it would also make sense to look for handshake records larger than 8kB (rough estimate).

This type of anomaly-centric intrusion detection is typically best done using the Bro IDS, which provides easy programmatic access to the X.509 certificate and SSL handshake.

There will be false positives when alerting on large certificates in this manner, which is why I recommend to also check if the certificates have been signed by a trusted root or not. A certificate that is signed by a trusted root is very unlikely to contain malicious data.

Posted by Erik Hjelmvik on Tuesday, 06 February 2018 12:13:00 (UTC/GMT)

Tags: #malware​ #C2​ #SSL​ #TLS​ #certificate​ #NetworkMiner​ #PCAP​ #x509​ #X.509​ #PIPI​ #Bro​ #IDS​ #tshark​