NETRESEC Network Security Blog - Tag : TLS Inspection

rss Google News

How to Inspect TLS Encrypted Traffic

Do you want to analyze decrypted TLS traffic in Wireshark or let an IDS, like Suricata, Snort or Zeek, inspect the application layer data of potentially malicious TLS encrypted traffic? There are many different TLS inspection solutions to choose from, but not all of them might be suitable for the specific challenge you’re facing. In this blog post I describe three different methods for decrypting TLS and explain when to use one or the other.

RSA Private Key TLS Key Log TLS Inspection Proxy
Works for all ciphers No (DHE cipher suites not supported) Yes Yes
TLS 1.3 supported No Yes Yes
Zero client configuration required Yes No (pre-master secrets must be logged or extracted from TLS libraries) No (root CA certificate must be installed)
Decrypts traffic from any application No (most applications use modern ciphers with forward secrecy, which RSA doesn’t provide) No (each method for TLS key extraction typically only supports a specific set of applications or TLS libraries) No (apps that use certificate pinning or a custom certificate trust store cannot be intercepted without patching the app)
L7 traffic in PCAP files can be analyzed without decrypting TLS No No Yes
Allows decrypted traffic to be mirrored to a network interface No No Yes

RSA Private Key

TLS decryption with a private RSA key was for a long time the preferred method for inspecting SSL and TLS traffic. This approach allowed anyone with access to the server’s private RSA key to decrypt the traffic and inspect the application layer (L7) communication.

The primary drawback with RSA private key decryption is that a stolen or leaked private RSA key can be used by an attacker to decrypt all previously captured traffic from that server, if RSA key exchange is being used. Modern TLS stacks have therefore deprecated such ciphers in favor of ones that support forward secrecy, which typically perform an ephemeral Diffie–Hellman (DHE) key exchange instead of reusing the same private RSA key over and over. This means that the RSA private key decryption method cannot be used if the client and server are using a key exchange algorithm that supports forward secrecy.

RSA private key decryption can only be performed when all these conditions are met:

  • The protocol version is SSL 3.0, TLS 1.0, TLS 1.1 or TLS 1.2 (RSA was removed in TLS 1.3)
  • The server has selected a cipher suite that use RSA key exchange, such as TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA or TLS_RSA_WITH_AES_128_CBC_SHA
  • The private key matches the server certificate (traffic cannot be decrypted with a client certificate or an intermediate or root certificate)
  • The session has not been resumed (the handshake must include a Client Key Exchange message)

This Wireshark display filter can be used to check if the server has selected an RSA cipher:

tls.handshake.type == 2 and tls.handshake.ciphersuite in {10,47,53,60,61,156,157}

You can check for a client key exchange message with:

tls.handshake.type == 16

A private RSA key can be loaded into Wireshark by clicking Edit, Preferences and RSA Keys. Another alternative is to use the command line tool tshark’s -ouat:rsa_keys switch like this:

tshark -r tls.pcap -ouat:rsa_keys:'"/path/rsa.key",""'

TLS Key Log

Wireshark can decrypt the TLS layer in captured network traffic if the pre-master secrets used to establish the encrypted connection are provided. These secrets, or encryption key material, can be loaded into Wireshark from an SSLKEYLOGFILE by clicking Edit, Preferences, Protocols, TLS, and setting the (Pre)-Master-Secret log filename to the path of your SSLKEYLOGFILE.

Wireshark SSLKEYLOGFILE

Another alternative is to encode the key material as metadata in a pcap-ng file with editcap like this:

editcap --inject-secrets tls,SSLKEYLOG.txt tls.pcap tls-and-keys.pcapng

The primary drawback with the TLS key log decryption method is that only Wireshark and tshark can be used to analyze the decrypted TLS traffic. You also need to get hold of the keys or pre-master secrets in order to perform the decryption. Some applications, such as Firefox, Chrome and curl, can be configured to export a key log. Another alternative is to install an agent that extracts key material from specific TLS libraries.

The limitation of only being able to extract keys from a specific set of applications or TLS libraries makes the TLS key log method unsuitable for analyzing TLS encrypted C2 traffic from malware, which often use custom TLS libraries. It is also difficult to send decrypted TLS traffic to an IDS or a network security monitoring tool using a TLS key log. If you, on the other hand, want to analyze network traffic from your own Firefox or Chrome browser in Wireshark, then the TLS key log approach is probably the best solution.

TLS Inspection Proxy

A TLS inspection proxy acts as a man-in-the-middle that intercepts and decrypts TLS traffic for inspection, it then re-encrypts the traffic and forwards it to the intended destination.

TLS inspection proxy

A major advantage of using a TLS inspection proxy is that decrypted TLS traffic can be analyzed from applications even if they use modern ciphers with forward secrecy and don’t support logging of TLS keys. The drawback, however, is that clients have to trust the root CA certificate that the proxy is using.

TLS inspection proxies often differ in how they make the decrypted traffic available to external tools, if at all. In fact, many TLS inspection proxies and Next-Generation Firewalls (NGFW) only make the decrypted payload available to the internal application or appliance. Such an approach prevents analysis of the decrypted traffic with an external tool, like Wireshark, Snort, Suricata, Zeek or NetworkMiner.

Another approach, used by proxies like mitmproxy, is to save a TLS key log for all proxied traffic. That approach allows captured TLS traffic to or from the proxy to be decrypted and inspected with Wireshark, but the application layer traffic cannot be inspected with other tools that don’t support TLS decryption using a key log.

The third and most integration friendly approach is to save the decrypted traffic in clear text form, so that other applications can inspect the unencrypted traffic without having to decrypt TLS. Some TLS proxies, like PolarProxy and SSLsplit, can even save the decrypted traffic to a PCAP file. Decrypted TLS traffic in PCAP format can easily be ingested into other tools or be replayed to a network interface for inspection by an external security appliance.

Best Practices

The list below can be used to select the best suited TLS inspection method for the particular challenge you’re tasked with.

I want to...

  • Inspect traffic from my browser.

    Use TLS key log to inspect traffic from Firefox, Chrome and curl. Use a TLS inspection proxy for other browsers.

  • Inspect traffic to my HTTPS website.

    Use RSA private key inspection if it is acceptable to use an older TLS version and less secure ciphers. Use a TLS key log if your web server can be configured to export one or if you have an agent based key extraction solution that supports the TLS library used by the web server. Use a TLS inspection proxy if you want to inspect the traffic with something other than Wireshark.

  • Inspect potentially malicious TLS traffic with an IDS or security appliance.

    Use a TLS inspection proxy.

  • Inspect traffic from my operating system.

    Use a TLS inspection proxy.

  • Inspect traffic from my mobile phone, smart device or other embedded device.

    Use a TLS inspection proxy.

  • Inspect traffic from a proprietary game, app or service.

    Use a TLS inspection proxy.

Posted by Erik Hjelmvik on Wednesday, 07 August 2024 11:40:00 (UTC/GMT)

Tags: #TLS#TLS Inspection#PolarProxy#SSLKEYLOGFILE#Wireshark#PCAP

Short URL: https://netresec.com/?b=248b1db


Sharing a PCAP with Decrypted HTTPS

Modern malware and botnet C2 protocols use TLS encryption in order to blend in with "normal" web traffic, sometimes even using legitimate services like Twitter or Instagram.

I did a live demo at the CS3Sthlm conference last year, titled "TLS Interception and Decryption", where I showed how TLS interception can be used to decrypt and analyze malicious HTTPS network traffic. During the demo I used DNS-over-HTTPS (DoH) and posted messages to Pastebin and Twitter, pretending to be a malware or malicious actor. The HTTPS network traffic was decrypted and analyzed live as part of my demo. The CS3Sthlm organizers have posted a video recording of the live demo on YouTube.

Erik presenting PolarProxy at CS3Sthlm, photo credit: CS3Sthlm

Image: Erik demoing TLS Interception and Decryption at CS3Sthlm 2019

We are now releasing a PCAP file with the decrypted network traffic captured during this live demo here:

» https://media.netresec.com/pcap/proxy-191023-091924.pcap «

This blog post provides a step-by-step walk-through of the decrypted HTTPS traffic in the released capture file.

The TLS decryption was performed by connecting a laptop to a custom WiFi access point, which was a Raspberry Pi configured as in our "Raspberry Pi WiFi Access Point with TLS Inspection" blog post. I additionally enabled the PCAP-over-IP feature in PolarProxy by starting it with the "--pcapoverip 57012" option. This allowed me to connect with Wireshark and NetworkMiner to TCP port 57012 on the TLS proxy and stream the decrypted traffic in order to perform live network traffic analysis.

Laptop, Raspberry Pi, PolarProxy, Internet ASCII

Image: Live demo network with Laptop (Browser, NetworkMiner, Wireshark), Raspberry Pi (PolarProxy) and the Internet.

Below follows a breakdown of various significant events of my demo and where you can find these events in the released capture file.

DNS lookup of "www.google.com" using DoH

  • Frame: 833
  • Protocol: DoH using HTTP/2 POST
  • Five tuple: 192.168.4.20:52694 104.16.248.249:80 TCP
DoH lookup of www.google.com shown in NetworkMiner DoH lookup of www.google.com shown in Wireshark

Google search for "tibetan fox psbattle"

  • Frame: 2292
  • Protocol: HTTP/2
  • Five tuple: 192.168.4.20:52716 216.58.211.4:80 TCP
Google search for 'tibetan fox psbattle' in Wireshark Google search for 'tibetan fox psbattle' in NetworkMiner

Tibetan Fox image downloaded from reddit

  • Frame: 3457
  • Protocol: HTTP/2
  • Five tuple: 192.168.4.20:52728 151.101.85.140:80 TCP
Image download from reddit shown in NetworkMiner

Orginal "tibetan fox" image downloaded from this reddit thread.

Tibetan Fox Remix Image HTTP/2 Download

  • Frame: 5805
  • Protocol: HTTP/2
  • Five tuple: 192.168.4.20:52769 151.101.84.193:80 TCP
Images downloaded via HTTP/2

DNS Lookup of "cs3sthlm.se"

  • Frame: 13494
  • Protocol: DoH using HTTP/2 POST
  • Five tuple: 192.168.4.20:52699 104.16.249.249:80 TCP

Images downloaded from CS3Sthlm's website

  • Frame: 14134
  • Protocol: HTTP/1.1
  • Five tuple: 192.168.4.20:52896 192.195.142.160:80 TCP
Images downloaded from CS3Sthlm's website

Data sent in HTTP/2 POST to Pastebin

  • Frame: 18572
  • Protocol: HTTP/2 POST
  • Five tuple: 192.168.4.20:52904 104.22.2.84:80 TCP
Data sent to Pastebin in HTTP/2 POST

The file "post.php.form-data" contains the data sent to Pastebin in the HTTP/2 POST request. Here are the reassembled contents of that file, including the "hello cs3 I am a malware" message:

-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="csrf_token_post"

MTU3MTgyMjg5OTFwcjBzODJaQ0NuUk9PT1B3ZTl0b20zdFg3ZkhXQ1R4
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="submit_hidden"

submit_hidden
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="paste_code"

hello cs3 I am a malware
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="paste_format"

1
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="paste_expire_date"

1H
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="paste_private"

0
-----------------------------54168074520069581482009826076
Content-Disposition: form-data; name="paste_name"

malware traffic
-----------------------------54168074520069581482009826076--

Mallory80756920 logs in to Twitter

  • Frame: 24881
  • Protocol: HTTP/2 POST
  • Five tuple: 192.168.4.20:53210 104.244.42.65:80 TCP
Twitter credentials for Mallory80756920

Mallory80756920 posts a Tweet

  • Frame: 26993
  • Protocol: HTTP/2 POST
  • Five tuple: 192.168.4.20:53251 104.244.42.66:80 TCP

Mallory80756920 tweeted "Hello CS3! I'm in you!". The data was sent to twitter using an HTTP/2 POST request.

Twitter post in Wireshark Twitter post in NetworkMiner

Conclusions

A great deal of the interesting TLS traffic in the analyzed capture file is using the HTTP/2 protocol. This doesn't come as a surprise since more than half of all HTTPS traffic is using HTTP/2 nowadays (sources: server protocol statistics, client protocol statistics). It is therefore essential to be able to analyze HTTP/2 traffic if you have a TLS inspection (TLSI) solution in place. Unfortunately many TLSI products don't yet support the HTTP/2 protocol.

Wireshark was one of the first network traffic analysis tools to implement HTTP/2 support, much thanks to Alexis La Goutte. However, Wireshark's excellent "File > Export Objects" doesn't yet support extraction of files from HTTP/2 traffic. There are other ways to extract HTTP/2 file transfers with Wireshark, but they require a few additional steps in order to carve out the file to disk.

Luckily NetworkMiner extracts files from HTTP/2 as of version 2.5. In fact, we believe NetworkMiner is the first open source tool to support automatic HTTP/2 file extraction from PCAP.

Finally, I'd like to stress the point that modern malware use HTTPS, so you need to have a TLSI solution in place to analyze the malicious traffic. As the majority of all HTTPS traffic is using HTTP/2 you also need to ensure that you're able to analyze HTTP/2 traffic passing through your TLSI solution.

Posted by Erik Hjelmvik on Monday, 13 January 2020 12:45:00 (UTC/GMT)

Tags: #HTTP/2#http2#DoH#TLS#Google#decrypt#HTTPS#TLSI#TLS Inspection#TLS Interception#PolarProxy#NetworkMiner#Wireshark#CS3Sthlm#CS3#Forensics#PCAP#Video

Short URL: https://netresec.com/?b=2015d89


The NSA HSTS Security Feature Mystery

NSA TLSI advisory header

I recently stumbled across an NSA Cyber Advisory titled Managing Risk from Transport Layer Security Inspection (U/OO/212028-19) after first learning about it through Jonas Lejon’s blog post NSA varnar för TLS-inspektion (Swedish). I read the NSA report with great interest since I wanted to see how our own TLS interception tool PolarProxy stands up to the risks identified in the advisory.

I highly respect the NSA’s knowledge in the fields of cryptography and security, which is why I read the advisory as if it was the definite guide on how to perform TLS inspection in a secure manner. However, towards the end of the advisory I got stuck on this paragraph, which I was unable to understand:

HTTP Strict Transport Security (HSTS) includes a security feature that binds the HTTP session to the specific TLS session used. TLSI systems that ignore the underlying HTTP headers will cause HSTS sessions to be rejected by the client application, the server, or both.

WUT?!?! HSTS (RFC 6797) is designed to protect against attacks like Moxie Marlinspike’s 10 year old sslstrip, which fools the client into using unencrypted HTTP instead of HTTPS. As far as I know, HSTS does not support binding an HTTP session to a specific TLS session.

What is HSTS?

HSTS is a simple HTTP header sent by the web server to inform the browser that it should only load the site using HTTPS. The header can look like this:

strict-transport-security: max-age=47111337; includeSubDomains; preload

NetworkMiner 2.5 showing HSTS headers from HTTP/1.1 and HTTP/2 traffic

Image: NetworkMiner 2.5 showing HSTS headers from HTTP/1.1 and HTTP/2 traffic

The PCAP file loaded into NetworkMiner in screenshot above contains HTTPS traffic that has been decrypted by PolarProxy. You can also use Wireshark or tshark to find HSTS headers by using the following display filter:

http.response.line matches "Strict-Transport-Security" or http2.header.name matches "Strict-Transport-Security"

Notice the use of “matches” to get case insensitive matching of “Strict-Transport-Security” and the duplicated statements required to get the filter to match headers in both HTTP/1.1 and HTTP/2.

Back to the NSA Advisory

I simply couldn’t understand NSA’s statement about the HSTS “security feature” that binds the HTTP session to the specific TLS session used, because I’m not aware of any such feature in HSTS and my google-fu was to weak to find any such feature. I therefore resorted to asking folks on Twitter if they knew about this feature.

Nick Sullivan (Head of Research and Cryptography at Cloudflare) kindly replied on twitter that they probably mean “Token Binding” (RFC 8471). Peter Wu (Wireshark core developer and TLS expert) additionally replied that the NSA perhaps were referring to the Sec-Token-Binding header name in RFC 8473 (which has nothing to do with HSTS).

Backed by these two experts in TLS and HTTPS I can confidently conclude that the authors of the NSA’s Transport Layer Security Inspection (TLSI) advisory have either misunderstood what HSTS is or made some form of typo. I have reported this error to the NSA, but have not yet received any response.

Other Issues in NSA’s TLSI Advisory

This experience dented my faith in the TLS expertise of the NSA in general and in particular the authors of this advisory. I therefore decided to re-read the TLSI advisory, but this time without the “NSA knows this stuff” glasses I had on before. This time I found several additional claims and statements, which I didn’t agree with, in the four page advisory.

To start with, the first figure in the document depicts two setups where one is allowing end-to-end encryption between the client and server, while the other is using TLS inspection.

Encrypted Traffic (top) and TLS Inspection (bottom) as shown in NSA’s TLSI advisory

Image: Encrypted Traffic (top) and TLS Inspection (bottom) as shown in NSA’s TLSI advisory

In the end-to-end setup we see the malicious traffic passing straight to the client, while in the second setup the malicious traffic is blocked by the TLSI device. This image is accompanied by the following text:

A TLSI capability implemented within a forward proxy between the edge of the enterprise network and an external network such as the Internet protects enterprise clients from the high risk environment outside the forward proxy.

My experience is that blocking traffic in this way is more likely to provide a false sense of security than improved protection against malware. It is better to use TLSI to detect intrusions than to try to block them.

This probably sounds counter-intuitive, as you might reason that “if you can detect it, why can't you prevent it?”. I’m not going to go into a lengthy explanation of the “Prevention Eventually Fails” paradigm here, instead I’d recommend reading Richard Bejtlichs book “The Tao of Network Security Monitoring” or his more recent book “The Practice of Network Security Monitoring”. You can also read Richards blog post about ”The Problem with Automated Defenses” or Phil Hagen’s “What to Do When Threat Prevention Fails”.

TLSI products should primarily be used to provide visibility in order to detect and respond to malware and intrusions. Many TLSI products can block some attacks, but they don’t really provide perimeter security. Unfortunately the NSA advisory might further strengthen this false sense of security with the mentioned image and accompanying text.

DO IT WELL, DO IT ONCE

The NSA advisory states:

To minimize the risks described above, breaking and inspecting TLS traffic should only be conducted once within the enterprise network. Redundant TLSI, wherein a client-server traffic flow is decrypted, inspected, and re-encrypted by one forward proxy and is then forwarded to a second forward proxy for more of the same, should not be performed.

In general, I agree with this statement. Unfortunately many TLSI products do not “do it well”, which is why “do it once” is not always enough. One such situation is when a PC is believed to be infected with malware, but further investigation is required to know for sure. Many TLSI products apply lock-in techniques that prevent incident responders from analyzing the decrypted traffic with external tools. This forces IR teams to add an extra TLS proxy when their enterprise TLSI product does not provide sufficient visibility. PolarProxy is, in fact, designed to support that type of agile TLSI deployment, in order to enable decryption and inspection of TLS traffic from a particular machine during an incident.

Final Words about the Advisory and TLS Inspection

Even through this blog post criticizes parts of the NSA TLSI advisory, I’d still like to end with saying that it also brings a lot of good stuff to the table. The issues mentioned in this blog post are actually just minor ones, which hopefully won’t have any negative impact on future TLSI deployments. In fact, the positive aspects of the advisory by far outweighs the negative ones. I also hope this blog post can help get more discussions going about TLS inspection, such as if we need it, what problems it can solve and what risks we take by deploying it.

UPDATE, 16 December 2019

The NSA released an updated version of the advisory today (1U/OO/212028-19). The advisory now says:

  • TLS Token Binding binds security tokens to the specific TLS session used. TLSI systems can cause the sessions or tokens to be rejected by the client application, the server, or both.
  • Hypertext Transfer Protocol Strict Transport Security (HSTS) requires that Hypertext Transfer Protocol over TLS (HTTPS) is used in the future with trusted certificates and that all content is received via HTTPS as well. If a TLS client application attempts to follow the HSTS requirements but does not trust the separate TLSI CA, the client will reject the TLSI sessions and prevent users from clicking-through browser warnings.

Thanks for updating!

Posted by Erik Hjelmvik on Tuesday, 26 November 2019 19:18:00 (UTC/GMT)

Tags: #TLSI#TLS Inspection#TLS Interception#inspect#PolarProxy#TLS#NetworkMiner#Wireshark

Short URL: https://netresec.com/?b=19B5cbd

X / twitter

X / Twitter: @netresec


Bluesky

Bluesky: @netresec.com


Mastodon

Mastodon: @netresec@infosec.exchange