NETRESEC Network Security Blog - Tag : HTTP/2

Capturing Decrypted TLS Traffic with Arkime

PolarProxy and Arkime Logo

The latest version of Arkime (The Sniffer Formerly Known As Moloch) can now be fed with a real-time stream of decrypted HTTPS traffic from PolarProxy. All that is needed to enable this feature is to include "pcapReadMethod=pcap-over-ip-server" in Arkime's config.ini file and start PolarProxy with the "--pcapoveripconnect" option. PolarProxy will then connect to Arkime's PCAP-over-IP listener on TCP port 57012 and send it a copy of all TLS packets it decrypts.

Note: The required PCAP-over-IP feature is available in Arkime 2.7.0 and PolarProxy 0.8.16.

About Arkime

Arkime is an open source packet capture solution that indexes the PCAP data it collects. Arkime also comes with a web frontend for browsing and searching through the captured, and indexed, network traffic. The Arkime project recently changed name from Moloch, probably in an attempt to convince users that the tool doesn't eat children.

How to Install Arkime with PolarProxy

This guide demonstrates how TLS traffic, or more specifically HTTPS traffic, can be decrypted and ingested in real-time into Arkime.

The TLS decryption is performed with PolarProxy, which is a transparent TLS interception proxy that is freely available under a Creative Commons BY-ND 4.0 license.

TLS decryption with PolarProxy and Arkime. TLS added and removed here.

PolarProxy and Arkime can be installed on a server to intercept, decrypt, index and store decrypted TLS network traffic from multiple clients on a network. It is even possible to install PolarProxy and Arkime on separate servers, so that PolarProxy forwards a stream of decrypted traffic to the Arkime server. However, to avoid unnecessary complexity, Arkime and PolarProxy are installed locally on a Linux client in this howto guide. The Linux client is a Ubuntu 20.04.1 machine, but the instructions can also be used on other Linux flavors that use systemd, such as Arch, CentOS, Debian, Fedora, SUSE and Red Hat Linux.

Download and Install Arkime

Arkime can be downloaded as a pre-built installation packages for CentOS and Ubuntu here:

Note: You can alternatively visit the Arkime GitHub page if there is no pre-built installation package for your Linux distro or you prefer to build Arkime from source.

After installing the Arkime package, configure Arkime by running:

sudo /data/moloch/bin/Configure
Found interfaces: lo;enp0s3 Semicolon ';' seperated list of interfaces to monitor [eth1] none
  • Enter "none" as the interface to monitor (the interface setting will be ignored when Arkime gets configured as a PCAP-over-IP server)
  • Install the ElasticSearch server by typing "yes" when prompted

Edit /data/moloch/etc/config.ini and add "pcapReadMethod=pcap-over-ip-server" to configure Arkime to listen for PCAP-over-IP connections.

pcapReadMethod=pcap-over-ip-server in Arkime's config.ini

Next, enable and start the ElasticSearch systemd service.

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Initiate the Arkime search cluster.

/data/moloch/db/ http://localhost:9200 init

Create a new admin user.

/data/moloch/bin/ admin "Admin User" THEPASSWORD --admin
Note: Feel free to pick a more secure password than "THEPASSWORD" for the admin user.

You can now enable and start the Moloch capture and viewer services.

sudo systemctl enable molochcapture.service
sudo systemctl start molochcapture.service
sudo systemctl enable molochviewer.service
sudo systemctl start molochviewer.service

Verify that Arkime now listens for incoming connections on TCP port 57012.

ss -nta | grep 57012
LISTEN 0 10*

Install PolarProxy to Decrypt TLS Traffic

Create a user for PolarProxy's systemd service and download PolarProxy like this:

sudo adduser --system --shell /bin/bash proxyuser
sudo mkdir /var/log/PolarProxy
sudo chown proxyuser:root /var/log/PolarProxy/
sudo chmod 0775 /var/log/PolarProxy/
sudo su - proxyuser
mkdir ~/PolarProxy
cd ~/PolarProxy/
curl | tar -xzf -

Copy the default PolarProxy service config to the systemd location.

sudo cp /home/proxyuser/PolarProxy/PolarProxy.service /etc/systemd/system/PolarProxy.service

Modify /etc/systemd/system/PolarProxy.service by adding "--pcapoveripconnect" at the end of the ExecStart command.

PolarProxy.service with --pcapoveripconnect

It's now time to enable and start the PolarProxy service.

sudo systemctl enable PolarProxy.service
sudo systemctl start PolarProxy.service

Verify that PolarProxy has connected to Arkime's PCAP-over-IP listener on TCP port 57012.

ss -nta | grep 57012
LISTEN 0 10*

Take it For a Test Run

PolarProxy is listening for incoming TLS connections on TCP port 10443. We can therefore run traffic through the TLS decryption proxy with this curl command:

curl --insecure --connect-to

The decrypted traffic will show up in Arkime if everything is working. Open http://localhost:8005/sessions in a browser and look for a connection to

Note: The Arkime username and password is admin/THEPASSWORD if you've followed the instructions in this tutorial.

Also: You might have to wait a minute or two for the traffic to appear in Arkime's user interface.

Moloch Sessions showing curl connection to

Trust PolarProxy's Root CA Certificate

The root CA certificate used by your PolarProxy service must be trusted by both the operating system and browser in order to run TLS traffic through the decryption proxy without errors. Follow these instructions to add trust the root CA:

sudo mkdir /usr/share/ca-certificates/extra
sudo openssl x509 -inform DER -in /var/log/PolarProxy/polarproxy.cer -out /usr/share/ca-certificates/extra/PolarProxy-root-CA.crt
sudo dpkg-reconfigure ca-certificates
  • Select the "extra/PolarProxy-root-CA.crt" Certificate Authority
  • Press <Ok>

Start Firefox

  • Download the root CA certificate from: http://localhost:10080/polarproxy.cer
  • Open: about:preferences#privacy
  • Scroll down to "Certificates" and click "View Certificates"
  • Import > Select "polarproxy.cer"
  • Select: ☑ Trust this CA to identify websites

Firefox: Trust this CA to identify websites

Configure Firewall Redirect of Outgoing HTTPS Traffic

The final step in this tutorial is to redirect the local user's outgoing HTTPS traffic to the PolarProxy service listening on TCP port 10443. Add the following lines at the top of /etc/ufw/before.rules (before the "*filter" section) to redirect outgoing HTTPS traffic to the local PolarProxy service listening on port 10443.

-A OUTPUT -m owner --uid 1000 -p tcp --dport 443 -j REDIRECT --to 10443

Firefox: Trust this CA to identify websites

Note: The UFW config in "before.rules" is equivalent to running "iptables -t nat -A OUTPUT -m owner --uid 1000 -p tcp --dport 443 -j REDIRECT --to 10443"

Make sure to modify the uid value (1000) in the firewall rule to match that of the local user that PolarProxy should decrypt the HTTPS traffic for. You can see your uid value by running the command "id -u". You can even redirect traffic from several users to PolarProxy, but it's important that you DON'T forward the outgoing HTTPS traffic from the "proxyuser" account. You will otherwise generate an infinite firewall redirect loop, where outgoing HTTPS traffic from PolarProxy is redirected back to PolarProxy again. You can check the proxyuser's uid with the command "id -u proxyuser".

After saving before.rules, reload UFW to activate the port redirection.

sudo ufw reload

Surf 'n' Snoop

Your Linux machine is now configured to send decrypted HTTPS traffic to Arkime for inspection. Open Firefox and visit some websites, then go back to Arkime and have a look at the traffic. Again, remember that there might be a few minutes' delay before the traffic appears in Arkime's user interface

HTTP/2 Session in Moloch

You'll probably notice that the majority of all HTTPS traffic is actually using the HTTP/2 protocol. Unfortunately Arkime's http2 support is still quite limited, but I'm hoping it will improve in future releases.

Luckily, both Wireshark and NetworkMiner (which runs fine in Linux by the way) can be used to parse and extract contents from HTTP/2 traffic. Just hit Arkime's "Download PCAP" button and open the capture file in a tool of your choice.

NetworkMiner 2.6 showing files ectracted from HTTP/2 traffic

Image: NetworkMiner in Linux with files extracted from decrypted HTTP/2 traffic

Posted by Erik Hjelmvik on Tuesday, 01 December 2020 07:50:00 (UTC/GMT)

Tags: #PolarProxy #TLS #HTTPS #decrypt #PCAP #systemd #systemctl #UFW #http2 #HTTP/2 #PCAP-over-IP #pcapoverip

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

NetworkMiner 2.6 Released

NetworkMiner 2.6

We are happy to announce the release of NetworkMiner 2.6 today! The network forensic tool is now even better at extracting emails, password hashes, FTP transfers and artifacts from HTTP and HTTP/2 traffic than before.

Some of the major improvements in this new release are related to extraction and presentation of emails from SMTP, POP3 and IMAP traffic. On that note, we’d like to thank Mandy van Oosterhout for reporting a bug in our email parser!

Emails extracted with NetworkMiner 2-6
Image: Emails extracted from SMTP and IMAP traffic

I have previously blogged about how to extract John-the-Ripper hashes from Kerberos network traffic with NetworkMiner. We have now added support for presenting LANMAN and NTLM credentials as JtR hashes as well.

NTLMv2 and Kerberos hashes in NetworkMiner 2.6
Image: JtR formatted NTLMv2 and Kerberos hashes in NetworkMiner 2.6

We have also improved NetworkMiner’s Linux support. Files, images and folders can now be opened in external tools directly from the NetworkMiner GUI also when running NetworkMiner in Linux using Mono 6 (or later). Linux users previously got a “System.ComponentModel.Win32Exception” error message saying something like “Cannot find the specified file” or “Access denied” due to a breaking change introduced in Mono version 6.

NetworkMiner running in Ubuntu 20.04
Image: NetworkMiner 2.6 running in Ubuntu 20.04 with Mono

The new release also comes with several updates of how HTTP and HTTP/2 traffic is handled and presented. We have, for example, added better extraction of data sent in HTTP (or HTTP/2) POST requests. Posted JSON formatted parameters are also extracted even if the JSON data has been gzip compressed. The “Accept-Language” header values in HTTP and HTTP/2 are extracted as “Host Details” in order to support forensic analysis of user language settings, as shown by Fox-IT in their “Operation Wocao - Shining a light on one of China’s hidden hacking groups” report.

NetworkMiner has supported decapsulation of tunneling protocols and protocols for network virtualization, like 802.1Q, GRE, PPPoE, VXLAN, OpenFlow, MPLS and EoMPLS, since version 2.1. We have now improved our GRE parser to also support NVGRE (RFC 7637) by adding support for Transparent Ethernet Bridging.

Jan Hesse sent us a feature request on Twitter earlier this year, where asked about support for FritzBox captures. We are happy to announce that NetworkMiner now supports the modified pcap format you get when sniffing network traffic with a FritzBox gateway.


NetworkMiner 2.6 can now also parse and extract SIP chat messages (RFC 3428) to the “Messages” tab. Audio extraction of VoIP calls is still a feature that is exclusively available only in NetworkMiner Professional though.

NetworkMiner Professional

Our commercial tool NetworkMiner Professional has received a few additional updates, such as support for analysis of HTTP/2 traffic in the “Browsers tab”. However, please note that NetworkMiner does not perform TLS decryption, so the HTTP/2 traffic will have to be decrypted by a TLS proxy like PolarProxy prior to being saved to a PCAP file.

HTTP/2 traffic in NetworkMiner Professional's Browsers tab

We have added a few new great online services to NetworkMiner Pro’s OSINT lookup as well, such as, Browserling, MalwareDomainList and VirusTotal lookups of URL’s in the “Browsers” tab. We have also added some additional external OSINT sources for lookups of IP addresses and domain names, such as MalwareDomainList and mnemonic ACT. The JA3 hash lookup menu in NetworkMiner Professional’s “Hosts” tab has also been extended to include GreyNoise.

URL lookup menu in NetworkMiner Professional's Browsers tab

NetworkMiner Pro previously played back G.722 VoIP audio at half speed. This issue has now been fixed, so that G.722 RTP audio is extracted and played back in 16k samples/s. The bug was due to an error in RFC 1890 that was later corrected in RFC 3551. Thanks to Michael "MiKa" Kafka for teaching us about this!

Excerpt from RFC 3551:

Even though the actual sampling rate for G.722 audio is 16,000 Hz, the RTP clock rate for the G722 payload format is 8,000 Hz because that value was erroneously assigned in RFC 1890 and must remain unchanged for backward compatibility. The octet rate or sample-pair rate is 8,000 Hz.

We’d also like to mention that NetworkMiner Professional now comes with improved analytical support to help investigators detect Tor traffic.

Upgrading to Version 2.6

Users who have purchased a license for NetworkMiner Professional 2.x can download a free update to version 2.6 from our customer portal, or use the “Help > Check for Updates” feature. Those who instead prefer to use the free and open source version can grab the latest version of NetworkMiner from the official NetworkMiner page.

Posted by Erik Hjelmvik on Wednesday, 23 September 2020 09:10:00 (UTC/GMT)

Tags: #NetworkMiner #SMTP #POP3 #IMAP #email #FTP #JtR #John #Mono #Linux #HTTP #HTTP/2 #GRE #SIP #VoIP #Tor #PCAP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Discovered Artifacts in Decrypted HTTPS

We released a PCAP file earlier this year, which was recorded as part of a live TLS decryption demo at the CS3Sthlm conference. The demo setup used PolarProxy running on a Raspberry Pi in order to decrypt all HTTPS traffic and save it in a PCAP file as unencrypted HTTP.

Laptop, Raspberry Pi, PolarProxy, Internet ASCII

This capture file was later used as a challenge for our twitter followers, when we made the following announcement:

The capture file released in this blog post contains a few interesting things that were captured unintentionally. Can you find anything strange, funny or unexpected in the pcap file? (1/2)

Followed by this message:

The person to submit the most interesting answer wins a “PCAP or it didn’t happen” t-shirt. Compete by including your discovery in a retweet or reply to this tweet, or in an email to info(at) We want your answers before the end of January. (2/2)

We'd like to thank everyone who submitted answers in this challenge, such as David Ledbetter, Christoffer Strömblad, RunΞ and Chris Sistrunk.

We're happy to announce that the winner of our challenge is David Ledbetter. Congratulations David!

So what were the interesting thing that could be found in the released capture file? Below is a short summary of some things that can be found.

Telemetry data sent to

A surprising amount of information about the Firefox browser was sent to, including things like:

  • Active browser addons
  • Active browser plugins
  • Firefox profile creation date
  • Browser search region
  • Default search engine
  • Regional locales
  • Screen width
  • Screen height
  • CPU vendor, family and model
  • HDD model, revision and type
  • Installed RAM
  • Operating system
  • Etc..

Here's an excerpt showing a part of the data sent to Mozilla:

"build": { "applicationId": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", "applicationName": "Firefox", "architecture": "x86-64", "buildId": "20191002194346", "version": "69.0.2", "vendor": "Mozilla", "displayVersion": "69.0.2", "platformVersion": "69.0.2", "xpcomAbi": "x86_64-gcc3", "updaterAvailable": false }, "partner": { "distributionId": "canonical", "distributionVersion": "1.0", "partnerId": "ubuntu", "distributor": "canonical", "distributorChannel": "ubuntu", "partnerNames": [ "ubuntu" ] }, "system": { "memoryMB": 3943, "virtualMaxMB": null, "cpu": { "count": 1, "cores": 1, "vendor": "GenuineIntel", "family": 6, "model": 42, "stepping": 7, "l2cacheKB": 256, "l3cacheKB": 4096, "speedMHz": null, "extensions": [ "hasMMX", "hasSSE", "hasSSE2", "hasSSE3", "hasSSSE3", "hasSSE4_1", "hasSSE4_2", "hasAVX", "hasAES" ] }, "os": { "name": "Linux", "version": "5.0.0-31-generic", "locale": "en-US" }, "hdd": { "profile": { "model": null, "revision": null, "type": null }, "binary": { "model": null, "revision": null, "type": null }, "system": { "model": null, "revision": null, "type": null } }, "gfx": { "D2DEnabled": null, "DWriteEnabled": null, "ContentBackend": "Skia", "Headless": false, "adapters": [ { "description": "llvmpipe (LLVM 8.0, 256 bits)", "vendorID": "0xffff", "deviceID": "0xffff", "subsysID": null, "RAM": 3942, "driver": null, "driverVendor": "mesa/llvmpipe", "driverVersion": "", "driverDate": null, "GPUActive": true } ], "monitors": [ { "screenWidth": 681, "screenHeight": 654 } ], "features": { "compositor": "basic", "gpuProcess": { "status": "unavailable" }, "wrQualified": { "status": "blocked-vendor-unsupported" }, "webrender": { "status": "opt-in" } } }, "appleModelId": null }, "settings": { "blocklistEnabled": true, "e10sEnabled": true, "e10sMultiProcesses": 8, "telemetryEnabled": false, "locale": "en-US", "intl": { "requestedLocales": [ "en-US" ], "availableLocales": [ "en-US", "en-CA", "en-GB" ], "appLocales": [ "en-US", "en-CA", "en-GB", "und" ], "systemLocales": [ "en-US" ], "regionalPrefsLocales": [ "sv-SE" ], "acceptLanguages": [ "en-US", "en" ] }, "update": { "channel": "release", "enabled": true, "autoDownload": false }, "userPrefs": { "browser.cache.disk.capacity": 1048576, "": "SE", "": false, "network.trr.mode": 2 }, "sandbox": { "effectiveContentProcessLevel": 4 }, "addonCompatibilityCheckEnabled": true, "isDefaultBrowser": false, "defaultSearchEngine": "google", "defaultSearchEngineData": { "name": "Google", "loadPath": "[distribution]/searchplugins/locale/en-US/google.xml", "origin": "default", "submissionURL": "" } }, "profile": { "creationDate": 18183, "firstUseDate": 18183 }

You can use the following Wireshark display filter to find all the data sent to Mozilla:

http.request.method eq POST and contains telemetry

Public IP Revealed in PCAP

The client's IP address was, which is part of the RFC 1918 192.168/16 private address space. It's therefore safe to assume that the client was behind a NAT (the client was in fact behind a double NAT). However, we noticed that the public IP of the client was revealed through multiple services in the captured network traffic. One of these services is the advertising exchange company AppNexus (, which sent the client's public IP address in an X-Proxy-Origin HTTP header.

X-Proxy-Origin HTTP header in Wireshark

You can use the following Wireshark/tshark display filter to find X-Proxy-Origin headers:

http.response.line matches "x-proxy-origin" or matches "x-proxy-origin"

We are using the "matches" operator here instead of "contains" or "==" because we want to perform case insensitive matching. You might also notice that we need a completely different display filter syntax to match HTTP/2 headers compared to what we are used to with HTTP/1.1.

Monty Python "Majestik møøse" reference in reddit x-header

The reddit server sends an HTTP/2 header called "x-moose" with a value of "majestic".

x-moose 1 : majestic header from reddit

This header refers to the opening credits of Monty Python and the Holy Grail.

Wi nøt trei a høliday in Sweden this yër?

Posted by Erik Hjelmvik on Tuesday, 17 March 2020 09:00:00 (UTC/GMT)

Tags: #HTTP/2 #http2 #TLS #decrypt #TLSI #PolarProxy #NetworkMiner #Wireshark #CS3Sthlm #CS3 #Forensics #PCAP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Reverse Proxy and TLS Termination

PolarProxy is primarily a TLS forward proxy, but it can also be used as a TLS termination proxy or reverse TLS proxy to intercept and decrypt incoming TLS traffic, such as HTTPS or IMAPS, before it is forwarded to a server. The proxied traffic can be accessed in decrypted form as a PCAP formatted data stream, which allows real-time analysis of the decrypted traffic by an IDS as well as post incident forensics with Wireshark.

PolarProxy version 0.8.15 and later can import an existing X.509 server certificate (aka leaf certificate or end-entity certificate) in order to perform the TLS decryption using a valid certificate signed by a trusted certificate authority. If no server certificate is provided, then PolarProxy falls back to generating server certificates on the fly and signing them with its own root CA certificate.

There are two principal ways to run PolarProxy as a reverse proxy, either as a TLS termination proxy or as a reverse proxy that decrypts and re-encrypts the traffic.

PolarProxy as a TLS Termination Proxy

TLS Termination Proxy

The TLS termination proxy mode is useful in order to offload the task of performing TLS encryption to PolarProxy instead of doing the decryption on the web server. This mode can also be used when the proxied services don’t support TLS encryption, such as legacy web servers or servers hosting other unencrypted services that you want to secure with TLS.

The following command sequence shows how to create a Let’s Encrypt SSL certificate, convert it to the PKCS#12 format, and load the server certificate into PolarProxy to terminate incoming HTTPS connections. In this setup PolarProxy decrypts the TLS traffic and relays the HTTP traffic to the web server on TCP port 80.

sudo certbot certonly --manual --preferred-challenges dns -d,

sudo openssl pkcs12 -export -out /etc/example.p12 -inkey /etc/letsencrypt/live/ -in /etc/letsencrypt/live/ --passout pass:PASSWORD

sudo mkdir /var/log/TlsTerminationProxy/

sudo ./PolarProxy --terminate --connect --nosni --servercert, -p 443,80,80 -o /var/log/TlsTerminationProxy/

Here’s a breakdown of the arguments sent to PolarProxy:

  • --terminate : Terminate incoming TLS sessions and forward proxied traffic in unencrypted form.
  • --connect : Forward all proxied traffic to instead of connecting to the host name provided in the SNI extension of the TLS ClientHello message.
  • --nosni : Treat incoming TLS sessions that don’t define a host name with the SNI extension as if they wanna to connect to “”.
  • --servercert, : Use the server certificate “/etc/example.p12” for incoming connections to “” and “”.
  • -p 443,80,80 : Listen on TCP port 443, save decrypted traffic in PCAP file as if it was directed to port 80, forward decrypted traffic to port 80.
  • -o /var/log/TlsTerminationProxy/ : Save decrypted traffic to hourly rotated PCAP files in “/var/log/TlsTerminationProxy/”.

PolarProxy is a generic TLS proxy that doesn’t care what application layer protocol the TLS tunnel carries. So if you want to terminate the TLS encryption of incoming IMAPS sessions as well, then simply append an additional argument saying “-p 993,143,143” to also forward decrypted IMAP sessions to This method can be used in order to wrap almost any TCP based protocol in a TLS tunnel, which can be useful for privacy reasons as well as to prevent network monitoring tools from detecting the actual application layer protocol.

PolarProxy as a Reverse TLS Proxy

Reverse TLS Proxy

There are setups for which it is preferable to also encrypt the internal sessions between PolarProxy and the final server. One such setup is when the server is hosting a web service with support for the HTTP/2 protocol, which in practice always uses TLS. Luckily PolarProxy is designed to decrypt and re-encrypt proxied traffic while also forwarding important TLS parameters, such as ALPN and SNI, between the internal and external TLS sessions.

To use TLS encryption on the inside as well as outside of PolarProxy, simply do as explained in the previous TLS termination section, but remove the “--terminate” argument and change the port argument to “-p 443,80,443” like this:

sudo ./PolarProxy --connect --nosni --servercert, -p 443,80,443 -o /var/log/ReverseTlsProxy/

PolarProxy will save the decrypted traffic as cleartext HTTP (or HTTP/2) to PCAP files in the “/var/log/ReverseTlsProxy/” directory.

Real-Time Analysis of Decrypted Traffic

Both the external (client-to-proxy) and internal (proxy-to-server) TCP sessions, in the reverse TLS proxy example above, are encrypted with TLS. This prevents passive network security monitoring tools, such as IDSs, DPI and DLP appliances, from analyzing the application layer data being sent and received. The PCAP files written to “/var/log/ReverseTlsProxy/” can be a valuable forensic asset when investigating an incident, but a real-time stream of the decrypted data is needed in order to swiftly detect and alert on potential security breaches and other incidents.

PolarProxy’s “--pcapoverip” option can be used to provide such a real-time stream of the decrypted data passing through the proxy. This data can easily be sent to a network interface using tcpreplay, as explained in our blog post “Sniffing Decrypted TLS Traffic with Security Onion”.

Security Considerations

The examples shown in this blog post all run PolarProxy with root privileges using sudo, which can be dangerous from a security perspective. PolarProxy is actually designed to be run without root privileges, but doing so prevents it from listening on a port below 1024. Luckily, this issue can easily be overcome with a simple port forwarding or redirect rule. The following iptables redirect rule can be used if PolarProxy is listening on TCP port 20443 and incoming HTTPS request are arriving to the eth0 interface of the proxy:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to 20443

PolarProxy does not support loading settings from a config file. The password for the PKCS12 certificate will therefore need to be supplied on the command line, which can make it visible from a process listing. If this is a concern for you, then please consider using “hidepid” to hide processes from other users. You can find instructions on how to use hidepid in hardening guides for Debian, Arch, SUSE and most other Linux flavors.

Posted by Erik Hjelmvik on Thursday, 12 March 2020 15:45:00 (UTC/GMT)

Tags: #PolarProxy #TLS #SSL #PCAP #decrypt #HTTPS #HTTP #HTTP/2 #http2 #IMAPS #decrypt

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Sharing a PCAP with Decrypted HTTPS

Modern malware and botnet C2 protocols use TLS encryption in order to blend in with "normal" web traffic, sometimes even using legitimate services like Twitter or Instagram.

I did a live demo at the CS3Sthlm conference last year, titled "TLS Interception and Decryption", where I showed how TLS interception can be used to decrypt and analyze malicious HTTPS network traffic. During the demo I used DNS-over-HTTPS (DoH) and posted messages to Pastebin and Twitter, pretending to be a malware or malicious actor. The HTTPS network traffic was decrypted and analyzed live as part of my demo. The CS3Sthlm organizers have posted a video recording of the live demo on YouTube.

Erik presenting PolarProxy at CS3Sthlm, photo credit: CS3Sthlm

Image: Erik demoing TLS Interception and Decryption at CS3Sthlm 2019

We are now releasing a PCAP file with the decrypted network traffic captured during this live demo here:

» «

This blog post provides a step-by-step walk-through of the decrypted HTTPS traffic in the released capture file.

The TLS decryption was performed by connecting a laptop to a custom WiFi access point, which was a Raspberry Pi configured as in our "Raspberry Pi WiFi Access Point with TLS Inspection" blog post. I additionally enabled the PCAP-over-IP feature in PolarProxy by starting it with the "--pcapoverip 57012" option. This allowed me to connect with Wireshark and NetworkMiner to TCP port 57012 on the TLS proxy and stream the decrypted traffic in order to perform live network traffic analysis.

Laptop, Raspberry Pi, PolarProxy, Internet ASCII

Image: Live demo network with Laptop (Browser, NetworkMiner, Wireshark), Raspberry Pi (PolarProxy) and the Internet.

Below follows a breakdown of various significant events of my demo and where you can find these events in the released capture file.

DNS lookup of "" using DoH

  • Frame: 833
  • Protocol: DoH using HTTP/2 POST
  • Five tuple: TCP
DoH lookup of shown in NetworkMiner DoH lookup of shown in Wireshark

Google search for "tibetan fox psbattle"

  • Frame: 2292
  • Protocol: HTTP/2
  • Five tuple: TCP
Google search for 'tibetan fox psbattle' in Wireshark Google search for 'tibetan fox psbattle' in NetworkMiner

Tibetan Fox image downloaded from reddit

  • Frame: 3457
  • Protocol: HTTP/2
  • Five tuple: TCP
Image download from reddit shown in NetworkMiner

Orginal "tibetan fox" image downloaded from this reddit thread.

Tibetan Fox Remix Image HTTP/2 Download

  • Frame: 5805
  • Protocol: HTTP/2
  • Five tuple: TCP
Images downloaded via HTTP/2

DNS Lookup of ""

  • Frame: 13494
  • Protocol: DoH using HTTP/2 POST
  • Five tuple: TCP

Images downloaded from CS3Sthlm's website

  • Frame: 14134
  • Protocol: HTTP/1.1
  • Five tuple: TCP
Images downloaded from CS3Sthlm's website

Data sent in HTTP/2 POST to Pastebin

  • Frame: 18572
  • Protocol: HTTP/2 POST
  • Five tuple: TCP
Data sent to Pastebin in HTTP/2 POST

The file "post.php.form-data" contains the data sent to Pastebin in the HTTP/2 POST request. Here are the reassembled contents of that file, including the "hello cs3 I am a malware" message:

Content-Disposition: form-data; name="csrf_token_post"

Content-Disposition: form-data; name="submit_hidden"

Content-Disposition: form-data; name="paste_code"

hello cs3 I am a malware
Content-Disposition: form-data; name="paste_format"

Content-Disposition: form-data; name="paste_expire_date"

Content-Disposition: form-data; name="paste_private"

Content-Disposition: form-data; name="paste_name"

malware traffic

Mallory80756920 logs in to Twitter

  • Frame: 24881
  • Protocol: HTTP/2 POST
  • Five tuple: TCP
Twitter credentials for Mallory80756920

Mallory80756920 posts a Tweet

  • Frame: 26993
  • Protocol: HTTP/2 POST
  • Five tuple: TCP

Mallory80756920 tweeted "Hello CS3! I'm in you!". The data was sent to twitter using an HTTP/2 POST request.

Twitter post in Wireshark Twitter post in NetworkMiner


A great deal of the interesting TLS traffic in the analyzed capture file is using the HTTP/2 protocol. This doesn't come as a surprise since more than half of all HTTPS traffic is using HTTP/2 nowadays (sources: server protocol statistics, client protocol statistics). It is therefore essential to be able to analyze HTTP/2 traffic if you have a TLS inspection (TLSI) solution in place. Unfortunately many TLSI products don't yet support the HTTP/2 protocol.

Wireshark was one of the first network traffic analysis tools to implement HTTP/2 support, much thanks to Alexis La Goutte. However, Wireshark's excellent "File > Export Objects" doesn't yet support extraction of files from HTTP/2 traffic. There are other ways to extract HTTP/2 file transfers with Wireshark, but they require a few additional steps in order to carve out the file to disk.

Luckily NetworkMiner extracts files from HTTP/2 as of version 2.5. In fact, we believe NetworkMiner is the first open source tool to support automatic HTTP/2 file extraction from PCAP.

Finally, I'd like to stress the point that modern malware use HTTPS, so you need to have a TLSI solution in place to analyze the malicious traffic. As the majority of all HTTPS traffic is using HTTP/2 you also need to ensure that you're able to analyze HTTP/2 traffic passing through your TLSI solution.

Posted by Erik Hjelmvik on Monday, 13 January 2020 12:45:00 (UTC/GMT)

Tags: #HTTP/2 #http2 #DoH #TLS #Google #decrypt #HTTPS #TLSI #TLS Inspection #TLS Interception #PolarProxy #NetworkMiner #Wireshark #CS3Sthlm #CS3 #Forensics #PCAP #Video

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

NetworkMiner 2.5 Released

NetworkMiner 2.5

I am happy to announce the release of NetworkMiner 2.5 today! This new version includes new features like JA3 and parsers for the HTTP/2 and DoH protocols. We have also added support for a few older protocols that are still widely used, such as Kerberos and the CIFS browser protocol. Additionally, NetworkMiner can now parse PCAP files up to twice as fast as before!

Improving Passive TLS Analysis with JA3

Almost all web traffic is TLS encrypted nowadays, which prevents incident responders, analysts and investigators from inspecting otherwise unencrypted HTTP traffic for clues about malicious behavior or criminal intent. This requires analysts to use alternative approaches, such as looking at hostnames and X.509 certificates. This type of analysis is supported by NetworkMiner, since it parses Server Name Indication fields in client TLS handshakes and extracts X.509 certificates automatically when PCAP files are loaded.

In this release we’ve also added support for another passive TLS analysis technique called JA3, which is a method for fingerprinting TLS client implementations.

NetworkMiner leverages the JA3 fingerprint database from Trisul Network Analytics in order to match observed JA3 hashes to hashes of known malware and “normal” applications. This is what it looks like when the capture file “snort.log.1428364808”, from the FIRST 2015 “Hands-on Network Forensics” training (available here), has been loaded into NetworkMiner 2.5:

JA3 fingerprint of a Skype client

Image: JA3 fingerprint of a Skype client in NetworkMiner 2.5

The JA3 hash is also available in the “Parameters” tab, which is useful in order to find out what hosts that particular TLS implementation was reaching out to.

Filtering on JA3 hash 06207a1730b5deeb207b0556e102ded2 in NetworkMiner 2.5

Image: Filtering on JA3 hash 06207a1730b5deeb207b0556e102ded2

HTTP/2 and DoH Support

Passive analysis of TLS traffic, such as HTTPS, often doesn’t give sufficient visibility. Many organizations therefore use TLS proxies in order to decrypt the traffic going in and out from their networks. However, more than half of all HTTPS traffic is actually http2 (RFC 7540) nowadays. This has previously been an issue for users who wanted to analyze decrypted http2 traffic from their TLS intercepting proxies with NetworkMiner. We’re happy to announce that NetworkMiner now can parse http2 traffic, that has been decrypted by a TLS proxy, and extract files from the http2 transfers.

NetworkMiner 2.5 also supports the DNS over HTTPS (DoH) protocol (RFC 8484), which is a technique for sending DNS queries as http2 POST requests and parsing the returned data as DNS responses. We’ve incorporated the DoH data into NetworkMiner’s DNS tab, so that you can analyze it just like normal DNS traffic.

DoH traffic to in NetworkMiner’s DNS tab

Image: DoH traffic to in NetworkMiner’s DNS tab

Please note that NetworkMiner 2.5 does not perform TLS decryption. This means that NetworkMiner can only parse the contents of a TLS stream if it has been decrypted by a TLS proxy, such as PolarProxy.

Extracting Kerberos Hashes from PCAP

NetworkMiner’s support for the Kerberos protocol allows you to passively track which user accounts that are authenticating to what services, simply by monitoring network traffic. This is a feature is essential in order to track credential theft and lateral movement by adversaries inside your networks. After implementing kerberos username and hash extraction we realized that this feature could also be valuable for penetration testers. We therefore decided to present extracted Kerberos credentials in a format that is compatible with tools like hashcat and John the Ripper.

Kerberos krb5pa, krb5asrep and krb5tgs credentials extracted from the Wireshark sample capture file

Image: Kerberos krb5pa, krb5asrep and krb5tgs credentials extracted from the Wireshark sample capture file Krb-contrained-delegation.cap

For more information about Kerberos hashes, please see our Extracting Kerberos Credentials from PCAP blog post.

Even more NetBIOS and CIFS Artifacts!

NetworkMiner is a popular tool for extracting files transferred over SMB and SMB2 from capture files. It can also extract a great deal of information about the communicating hosts from protocols like NetBIOS and SMB/CIFS, but earlier this year Chris Raiter notified us about an important piece of information that was missing in NetworkMiner: NetBIOS Name Service (NBNS) lookups and responses!

Detection and export of NBNS packets request on twitter

A couple of months later Dan Gunter sent us another great feature request for another protocol that runs on top of NetBIOS: the CIFS Browser Protocol (aka MS-BRWS).

We’re happy to announce that NBNS queries and responses are now shown in NetworkMiner’s Parameters tab, and details like hostnames, domain names, Windows versions and uptime us extracted from the MS-BRWS protocol. See the screenshots below, which were created by loading the capture file “case09.pcap” from Richard Bejtlich’s TCP/IP Weapons School 2.0 Sample Lab into NetworkMiner 2.5. Thanks for sharing Richard!

Hostname, domain and Windows version extracted from MS-BRWS traffic

Image: Hostname, domain and Windows version extracted from MS-BRWS traffic

NBNS queries and responses in NetworkMiner’s Parameters tab

Image: NBNS queries and responses in NetworkMiner’s Parameters tab

Mono 5 Required for Linux and MacOS

Linux and MacOS users, who run NetworkMiner with help of Mono, will need to ensure they have Mono 5 (or later) installed in order to run NetworkMiner 2.5. We recommend using at least Mono 5.18.

Instructions for installing NetworkMiner on Linux can be found in our blog post ”HowTo install NetworkMiner in Ubuntu Fedora and Arch Linux”.

MacOS users can refer to our “Running NetworkMiner on Mac OS X” blog post.

Users who are unable to install Mono 5 are recommended to use the old NetworkMiner 2.4 release, which can be downloaded here:

NetworkMiner Professional

Apart from the features mentioned so far, our commercial tool NetworkMiner Professional now comes with a few additional new features. One of these features is port independent identification of RDP traffic, so that mstshash credentials can be extracted from RDP sessions even if the service doesn’t run on port 3389. The OSINT lookup context menus in NetworkMiner Professional have also been enriched with the following online services:

Several new features have also been included in the command line tool NetworkMinerCLI, including:

  • Recursive loading of PCAP files with the "-R” switch.
  • Configurable export types (hosts, files, DNS etc) with the “-x” switch.
  • Relative paths in CSV, XML and JSON/CASE exports unless the “-- absolutePaths” switch is used.


I’d like to thank Dan Gunter, Chris Raiter, Chris Sistrunk and a few more (who I cannot mention here) for contributing with feature requests and bug reports that have helped improve NetworkMiner.

Upgrading to Version 2.5

Users who have purchased a license for NetworkMiner Professional 2.x can download a free update to version 2.5 from our customer portal, or use the “Help > Check for Updates” feature. Those who instead prefer to use the free and open source version can grab the latest version of NetworkMiner from the official NetworkMiner page.

Posted by Erik Hjelmvik on Thursday, 07 November 2019 11:45:00 (UTC/GMT)

Tags: #NetworkMiner #HTTP/2 #http2 #DoH #Kerberos #NetBIOS #PCAP #hashcat #John #NetworkMinerCLI #OSINT

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

Raspberry PI WiFi Access Point with TLS Inspection

This is a how-to guide for setting up a Raspberry Pi as a WiFi Access Point, which acts as a transparent TLS proxy and saves the decrypted traffic in PCAP files.

Raspberry Pi 4 Model B running PolarProxy
Image: Raspberry Pi 4 Model B running PolarProxy

Step 1: Install PolarProxy for Linux ARM

We will start with installing PolarProxy, which will be used for the TLS decryption and re-encryption. The steps are almost identical to those in the official PolarProxy installation guide, except here we will download the "linux-arm" build of PolarProxy instead of the x64 version.

sudo adduser --system --shell /bin/bash proxyuser
sudo mkdir /var/log/PolarProxy
sudo chown proxyuser:root /var/log/PolarProxy/
sudo chmod 0775 /var/log/PolarProxy/
sudo su - proxyuser
mkdir ~/PolarProxy
cd ~/PolarProxy/
curl | tar -xzf -
sudo cp /home/proxyuser/PolarProxy/PolarProxy.service /etc/systemd/system/PolarProxy.service
sudo systemctl enable PolarProxy.service
sudo systemctl start PolarProxy.service

Note: The installation instructions above will fail on 64-bit ARM Linux OS's since the downloaded PolarProxy tarball is compiled for 32-bit ARM Linux. Luckily, we do have a linux-arm64 build as well, which is available here:

Verify that the PolarProxy service is running as expected with these commands:

systemctl status PolarProxy.service
journalctl -t PolarProxy

Step 2: Set up your Pi as a WiFi AP

The Raspberry Pi Foundation have a great guide for "Setting up a Raspberry Pi as a Wireless Access Point". Follow the instructions in their guide for the NAT mode setup (first section), but replace the iptables config with this:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A INPUT -i wlan0 -p tcp --dport 10443 -m state --state NEW -j ACCEPT
sudo iptables -t nat -A PREROUTING -i wlan0 -p tcp --dport 443 -j REDIRECT --to 10443
Then save the iptables rules with:
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"
Finally, edit /etc/rc.local and add this iptables-restore command just above "exit 0" to install the rules on boot.
iptables-restore < /etc/iptables.ipv4.nat

Step 3: Configure the Clients

The final step is to connect the clients (phones, tablets or computers) to the Raspberry Pi WiFi Access Point and install the root CA from PolarProxy.

Follow the instructions for "Trusting the PolarProxy root CA" in the official PolarProxy setup guide to install the public certificate from the TLS proxy in your clients. The certificate can be downloaded from the Raspberry Pi by browsing to


Your Raspberry Pi WiFi AP will now intercept all HTTPS traffic going to tcp/443 and save the decrypted traffic in PCAP files, one per hour. The PCAP files with decrypted TLS traffic can be found in the /var/log/PolarProxy/ directory of your Raspberry Pi.

pi@raspberrypi:/var/log/PolarProxy $ ls *.pcap
proxy-190925-075704.pcap proxy-190925-152902.pcap
proxy-190925-085704.pcap proxy-190925-162902.pcap
proxy-190925-095704.pcap proxy-190925-172902.pcap
proxy-190925-105704.pcap proxy-190925-182902.pcap
proxy-190925-115704.pcap proxy-190926-062902.pcap
proxy-190925-125704.pcap proxy-190926-072902.pcap
proxy-190925-132704.pcap proxy-190926-082902.pcap
proxy-190925-132902.pcap proxy-190926-092902.pcap
proxy-190925-142902.pcap proxy-190926-102902.pcap

HTTP/2 traffic to Facebook opened in Wireshark
Image: Decrypted HTTP/2 traffic to Facebook opened in Wireshark

Posted by Erik Hjelmvik on Thursday, 26 September 2019 11:37:00 (UTC/GMT)

Tags: #PolarProxy #PCAP #WiFi #TLS #SSL #HTTPS #Wireshark #http2

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:

PolarProxy Released

I’m very proud to announce the release of PolarProxy today! PolarProxy is a transparent TLS proxy that decrypts and re-encrypts TLS traffic while also generating a PCAP file containing the decrypted traffic.

PolarProxy flow chart

PolarProxy enables you to do lots of things that have previously been impossible, or at least very complex, such as:

  • Analyzing HTTP/2 traffic without an SSLKEYLOGFILE
  • Viewing decrypted HTTPS traffic in real-time using Wireshark
    PolarProxy -p 10443,80,443 -w - | wireshark -i - -k
  • Replaying decrypted traffic to an internal or external interface using tcpreplay
    PolarProxy -p 10443,80,443 -w - | tcpreplay -i eth1 -
  • Forwarding of decrypted traffic to a NIDS (see tcpreplay command above)
  • Extracting DNS queries and replies from DNS-over-TLS (DoT) or DNS-over-HTTPS (DoH) traffic
    PolarProxy -p 853,53 -p 443,80 -w dns.pcap
  • Extracting email traffic from SMTPS, POP3S or IMAPS
    PolarProxy -p 465,25 -p 995,110 -p 993,143 -w emails.pcap

Here is an example PCAP file generated by PolarProxy:

This capture files contains HTTP, WebSocket and HTTP/2 packets to Mozilla, Google and Twitter that would otherwise have been encrypted with TLS.

 HTTP/2 traffic from PolarProxy opened in Wireshark
Image: HTTP/2 traffic from PolarProxy opened in Wireshark

Now, head over to our PolarProxy page and try it for yourself (it’s free)!

Posted by Erik Hjelmvik on Friday, 21 June 2019 06:00:00 (UTC/GMT)

Tags: #PolarProxy #PCAP #NIDS #IDS #http2 #HTTP/2 #Wireshark #tcpreplay #DoH #SMTPS #IMAPS #TLS #SSL

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL:


NETRESEC on Twitter

Follow @netresec on twitter:


Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)