NETRESEC Network Security Blog - Tag : TLS

rss Google News

PolarProxy in Windows Sandbox

In this video I demonstrate how PolarProxy can be run in a Windows Sandbox to intercept and decrypt outgoing TLS communication. This setup can be used to inspect otherwise encrypted traffic from malware or suspicious Windows applications, which communicate over HTTPS or some other TLS encrypted protocol.

The Windows Sandbox WSB file used in the demo can be downloaded from here: https://www.netresec.com/?download=PolarProxySandbox

Note: Windows Pro or Enterprise is required to run WSB files

Parsing Decrypted TLS Traffic with NetworkMiner

This sandbox also includes NetworkMiner, primarily because it can be used to read a real-time PCAP-over-IP stream with decrypted traffic from PolarProxy. As shown in the video, this feature can be used in order to extract files, images or parameters from the decrypted TLS traffic in near real-time.

Images extracted from decrypted HTTP/2 traffic shown in NetworkMiner

For more info about how to run NetworkMiner in Windows Sandbox, please see our blog post Running NetworkMiner in Windows Sandbox.

Configuring a Proxy Server in Windows Sandbox

Windows’ built-in proxy settings are unfortunately not available in Windows Sandbox, which is why I installed a third-party proxy client that redirects all outgoing network traffic to PolarProxy’s SOCKS server. I used Proxifier in the video, which has the additional benefit of being able to redirect all traffic to the proxy, even from applications that aren’t proxy aware. This feature is crucial when attempting to intercept and decrypt TLS traffic from malware that doesn’t respect the proxy settings configured in the operating system.

Command Log

Start PolarProxy with a PCAP-over-IP listener on TCP 57012, SOCKS server on TCP 1080, HTTP proxy on 8080 and a transparent TLS proxy on port 443:

PolarProxy --pcapoverip 57012 -x ..\proxyroot.cer --socks 1080 --httpconnect 8080 --allownontls -p 443,80

Test PolarProxy’s SOCKS server by sending an unencrypted HTTP request through the proxy:

curl --socks4 localhost http://www.netresec.com

Test PolarProxy’s SOCKS server by sending an HTTPS request through the proxy:

curl --insecure --socks4 localhost https://www.netresec.com

Test PolarProxy’s HTTP CONNECT proxy server by sending an HTTPS request through the proxy:

curl --insecure --proxy localhost:8080 https://www.netresec.com

Start Menu Search

As shown in the video, text typed into Windows’ start menu gets sent to Microsoft. For more information about this behavior, and how it can be disabled, check out our Start Menu Search video and blog post.

Posted by Erik Hjelmvik on Monday, 31 January 2022 09:50:00 (UTC/GMT)

Tags: #PolarProxy#NetworkMiner#SOCKS#proxy#Windows Sandbox#Sandbox#PCAP-over-IP#pcapoverip#Windows#TLS#HTTPS

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=221d46b


PolarProxy 0.9 Released

PolarProxy 0.9

PolarProxy was previously designed to only run as a transparent TLS proxy. But due to popular demand we’ve now extended PolarProxy to also include a SOCKS proxy and a HTTP CONNECT proxy. PolarProxy automatically decrypts all proxied SSL and TLS traffic, regardless if the remote server is running on TCP 443 or some other port, as long as the traffic passes through PolarProxy. As from now we also release a Windows build of PolarProxy, alongside the Linux x64, ARM and ARM64 builds.

SOCKS Proxy

Use the command line argument “--socks [port]” to start PolarProxy’s SOCKS proxy server. This SOCKS proxy supports multiple versions of the SOCKS protocol, including SOCKS 4, SOCKS 4a, SOCKS 5 and SOCKS 5h.

As an example, the command below starts a SOCKS server on TCP port 1080 and passes a copy of the decrypted TLS traffic as a PCAP stream to tshark.

PolarProxy --socks 1080 -w - | tshark -r - -d tcp.port==443,http2
Note: The “-d tcp.port==443,http2” argument in the command above is used to tell tshark to parse traffic to port 443 as HTTP/2 instead of TLS. An alternative method would be to instead configure PolarProxy to output decrypted 443 traffic as if it was port 80, by supplying the “-p 443,80” argument to PolarProxy.

You can then use curl to run some HTTPS traffic through the SOCKS proxy:

curl --insecure --socks4 localhost https://www.netresec.com

After doing this you should see the decrypted HTTP/2 traffic in tshark’s output.

HTTP CONNECT Proxy

We’ve also added a HTTP proxy to PolarProxy 0.9, but it only supports the CONNECT request method. This means that normal unencrypted HTTP requests, like GET or POST requests, will be rejected by PolarProxy. Most web traffic is TLS encrypted nowadays anyway, so we don't consider this limitation to be a big issue.

The HTTP CONNECT proxy service is activated with the “--httpconnect” argument. Decrypted TLS traffic from PolarProxy’s HTTP CONNECT proxy can be forwarded to tshark just like in the SOCKS example, but the traffic from these proxies can also be accessed through PCAP-over-IP like this:

PolarProxy --httpconnect 8080 -p 443,80 --pcapoverip 57012

You can then connect to PolarProxy’s PCAP-over-IP service with NetworkMiner by clicking File, Receive PCAP over IP, select “Connect to IP/port”, enter “localhost” and click the “Start Receiving” button. You’ll now be able to see a real-time feed of all the traffic that PolarProxy decrypts. As an example, let’s download the PolarProxy logo over HTTPS to see if NetworkMiner can extract it from PolarProxy’s decrypted PCAP-over-IP stream:

curl --insecure --proxy localhost:8080 https://www.netresec.com/images/PolarProxy_313x313.png

The PolarProxy logo immediately shows up in NetworkMiner’s images tab:

NetworkMiner reading PCAP-over-IP from PolarProxy

Port-Independent TLS Protocol Detection

When PolarProxy is running as a transparent TLS proxy all incoming traffic can be expected to be TLS. But that’s not the case when, for example, PolarProxy is running as a SOCKS proxy. We have therefore added port-independent TLS protocol detection for proxied traffic, so that TLS traffic can be detected and decrypted even when it runs on other ports than the standard 443, 465, 853, 990, 993, 995 and 5061 ones.

There is one crucial limitation to the automatic SSL/TLS protocol detection though, it doesn’t support explicit TLS traffic that relies on opportunistic encryption features like STARTTLS, which bootstraps TLS into an already established application layer session.

Allow Non-TLS Traffic

SOCKS and HTTP CONNECT proxies can both be used to transport other protocols than TLS. PolarProxy blocks all non-TLS traffic by default, but this setting can be overridden with the “--allownontls” argument to allow any traffic to be proxied. The allow non-TLS override has no effect on PolarProxy’s transparent proxy though, because it will need to see a valid SNI field in order to know whereto the traffic should be forwarded.

Windows Build

There wasn’t much need for a Windows build of PolarProxy prior to the release of version 0.9, because the Windows firewall can’t be configured to redirect outgoing port 443 traffic to a local service. However, now that PolarProxy also includes SOCKS and HTTP CONNECT services, the situation is completely different. There are many ways to configure a Windows PC, as well as web browsers and other applications, to use a local proxy server.

You can use the Proxy settings window in Windows 10 and 11 to enable a local HTTP proxy like this:

Windows 10 Proxy Settings

Another option is to run “inetcpl.cpl” (Internet Options), open the “Connections” tab and click the “LAN settings” button to configure an HTTP proxy.

Windows Internet Options LAN Proxy Settings

You can, of course, also configure your browser to use a local SOCKS or HTTP proxy in Windows, just as you’d do on any other operating system.

But don’t forget to configure your OS and/or browser to trust your PolarProxy instance’s root CA certificate first, as explained in the “Trusting the PolarProxy root CA” section of our PolarProxy documentation.

The Windows version of PolarProxy is a .NET framework-dependent application, which requires the .NET 6 runtime to be installed. The PolarProxy releases for other platforms (Linux x64, ARM and ARM64) are all self-contained applications, which are published with the .NET runtime built-in.

Visit our PolarProxy page to download and install PolarProxy.

Posted by Erik Hjelmvik on Thursday, 13 January 2022 10:15:00 (UTC/GMT)

Tags: #PolarProxy#proxy#SOCKS#TLS#SSL#decrypt#Windows#PCAP-over-IP#pcapoverip

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=221953b


Capturing Decrypted TLS Traffic with Arkime

PolarProxy and Arkime Logo

The latest version of Arkime (The Sniffer Formerly Known As Moloch) can now be fed with a real-time stream of decrypted HTTPS traffic from PolarProxy. All that is needed to enable this feature is to include "pcapReadMethod=pcap-over-ip-server" in Arkime's config.ini file and start PolarProxy with the "--pcapoveripconnect 127.0.0.1:57012" option. PolarProxy will then connect to Arkime's PCAP-over-IP listener on TCP port 57012 and send it a copy of all TLS packets it decrypts.

Note: The required PCAP-over-IP feature is available in Arkime 2.7.0 and PolarProxy 0.8.16.

About Arkime

Arkime is an open source packet capture solution that indexes the PCAP data it collects. Arkime also comes with a web frontend for browsing and searching through the captured, and indexed, network traffic. The Arkime project recently changed name from Moloch, probably in an attempt to convince users that the tool doesn't eat children.

How to Install Arkime with PolarProxy

This guide demonstrates how TLS traffic, or more specifically HTTPS traffic, can be decrypted and ingested in real-time into Arkime.

The TLS decryption is performed with PolarProxy, which is a transparent TLS interception proxy that is freely available under a Creative Commons BY-ND 4.0 license.

TLS decryption with PolarProxy and Arkime. TLS added and removed here.

PolarProxy and Arkime can be installed on a server to intercept, decrypt, index and store decrypted TLS network traffic from multiple clients on a network. It is even possible to install PolarProxy and Arkime on separate servers, so that PolarProxy forwards a stream of decrypted traffic to the Arkime server. However, to avoid unnecessary complexity, Arkime and PolarProxy are installed locally on a Linux client in this howto guide. The Linux client is a Ubuntu 20.04.1 machine, but the instructions can also be used on other Linux flavors that use systemd, such as Arch, CentOS, Debian, Fedora, SUSE and Red Hat Linux.

Download and Install Arkime

Arkime can be downloaded as a pre-built installation packages for CentOS and Ubuntu here: https://arkime.com/#download

Note: You can alternatively visit the Arkime GitHub page if there is no pre-built installation package for your Linux distro or you prefer to build Arkime from source.

After installing the Arkime package, configure Arkime by running:

sudo /data/moloch/bin/Configure
Found interfaces: lo;enp0s3 Semicolon ';' seperated list of interfaces to monitor [eth1] none
  • Enter "none" as the interface to monitor (the interface setting will be ignored when Arkime gets configured as a PCAP-over-IP server)
  • Install the ElasticSearch server by typing "yes" when prompted

Edit /data/moloch/etc/config.ini and add "pcapReadMethod=pcap-over-ip-server" to configure Arkime to listen for PCAP-over-IP connections.

pcapReadMethod=pcap-over-ip-server in Arkime's config.ini

Next, enable and start the ElasticSearch systemd service.

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Initiate the Arkime search cluster.

/data/moloch/db/db.pl http://localhost:9200 init

Create a new admin user.

/data/moloch/bin/moloch_add_user.sh admin "Admin User" THEPASSWORD --admin
Note: Feel free to pick a more secure password than "THEPASSWORD" for the admin user.

You can now enable and start the Moloch capture and viewer services.

sudo systemctl enable molochcapture.service
sudo systemctl start molochcapture.service
sudo systemctl enable molochviewer.service
sudo systemctl start molochviewer.service

Verify that Arkime now listens for incoming connections on TCP port 57012.

ss -nta | grep 57012
LISTEN 0 10 0.0.0.0:57012 0.0.0.0:*

Install PolarProxy to Decrypt TLS Traffic

Create a user for PolarProxy's systemd service and download PolarProxy like this:

sudo adduser --system --shell /bin/bash proxyuser
sudo mkdir /var/log/PolarProxy
sudo chown proxyuser:root /var/log/PolarProxy/
sudo chmod 0775 /var/log/PolarProxy/
sudo su - proxyuser
mkdir ~/PolarProxy
cd ~/PolarProxy/
curl https://www.netresec.com/?download=PolarProxy | tar -xzf -
exit

Copy the default PolarProxy service config to the systemd location.

sudo cp /home/proxyuser/PolarProxy/PolarProxy.service /etc/systemd/system/PolarProxy.service

Modify /etc/systemd/system/PolarProxy.service by adding "--pcapoveripconnect 127.0.0.1:57012" at the end of the ExecStart command.

PolarProxy.service with --pcapoveripconnect 127.0.0.1:57012

It's now time to enable and start the PolarProxy service.

sudo systemctl enable PolarProxy.service
sudo systemctl start PolarProxy.service

Verify that PolarProxy has connected to Arkime's PCAP-over-IP listener on TCP port 57012.

ss -nta | grep 57012
LISTEN 0 10 0.0.0.0:57012 0.0.0.0:*
ESTAB 0 0 127.0.0.1:40801 127.0.0.1:57012
ESTAB 0 0 127.0.0.1:57012 127.0.0.1:40801

Take it For a Test Run

PolarProxy is listening for incoming TLS connections on TCP port 10443. We can therefore run traffic through the TLS decryption proxy with this curl command:

curl --insecure --connect-to www.netresec.com:443:127.0.0.1:10443 https://www.netresec.com/

The decrypted traffic will show up in Arkime if everything is working. Open http://localhost:8005/sessions in a browser and look for a connection to www.netresec.com.

Note: The Arkime username and password is admin/THEPASSWORD if you've followed the instructions in this tutorial.

Also: You might have to wait a minute or two for the traffic to appear in Arkime's user interface.

Moloch Sessions showing curl connection to www.netresec.com

Trust PolarProxy's Root CA Certificate

The root CA certificate used by your PolarProxy service must be trusted by both the operating system and browser in order to run TLS traffic through the decryption proxy without errors. Follow these instructions to add trust the root CA:

sudo mkdir /usr/share/ca-certificates/extra
sudo openssl x509 -inform DER -in /var/log/PolarProxy/polarproxy.cer -out /usr/share/ca-certificates/extra/PolarProxy-root-CA.crt
sudo dpkg-reconfigure ca-certificates
  • Select the "extra/PolarProxy-root-CA.crt" Certificate Authority
  • Press <Ok>

Start Firefox

  • Download the root CA certificate from: http://localhost:10080/polarproxy.cer
  • Open: about:preferences#privacy
  • Scroll down to "Certificates" and click "View Certificates"
  • Import > Select "polarproxy.cer"
  • Select: ☑ Trust this CA to identify websites

Firefox: Trust this CA to identify websites

Configure Firewall Redirect of Outgoing HTTPS Traffic

The final step in this tutorial is to redirect the local user's outgoing HTTPS traffic to the PolarProxy service listening on TCP port 10443. Add the following lines at the top of /etc/ufw/before.rules (before the "*filter" section) to redirect outgoing HTTPS traffic to the local PolarProxy service listening on port 10443.

*nat
:OUTPUT ACCEPT [0:0]
-A OUTPUT -m owner --uid 1000 -p tcp --dport 443 -j REDIRECT --to 10443
COMMIT

Firefox: Trust this CA to identify websites

Note: The UFW config in "before.rules" is equivalent to running "iptables -t nat -A OUTPUT -m owner --uid 1000 -p tcp --dport 443 -j REDIRECT --to 10443"

Make sure to modify the uid value (1000) in the firewall rule to match that of the local user that PolarProxy should decrypt the HTTPS traffic for. You can see your uid value by running the command "id -u". You can even redirect traffic from several users to PolarProxy, but it's important that you DON'T forward the outgoing HTTPS traffic from the "proxyuser" account. You will otherwise generate an infinite firewall redirect loop, where outgoing HTTPS traffic from PolarProxy is redirected back to PolarProxy again. You can check the proxyuser's uid with the command "id -u proxyuser".

After saving before.rules, reload UFW to activate the port redirection.

sudo ufw reload

Surf 'n' Snoop

Your Linux machine is now configured to send decrypted HTTPS traffic to Arkime for inspection. Open Firefox and visit some websites, then go back to Arkime and have a look at the traffic. Again, remember that there might be a few minutes' delay before the traffic appears in Arkime's user interface

HTTP/2 Session in Moloch

You'll probably notice that the majority of all HTTPS traffic is actually using the HTTP/2 protocol. Unfortunately Arkime's http2 support is still quite limited, but I'm hoping it will improve in future releases.

Luckily, both Wireshark and NetworkMiner (which runs fine in Linux by the way) can be used to parse and extract contents from HTTP/2 traffic. Just hit Arkime's "Download PCAP" button and open the capture file in a tool of your choice.

NetworkMiner 2.6 showing files ectracted from HTTP/2 traffic

Image: NetworkMiner in Linux with files extracted from decrypted HTTP/2 traffic

Posted by Erik Hjelmvik on Tuesday, 01 December 2020 07:50:00 (UTC/GMT)

Tags: #PolarProxy#TLS#HTTPS#decrypt#PCAP#systemd#systemctl#UFW#http2#HTTP/2#PCAP-over-IP#pcapoverip#ASCII-art

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=20C3247


PolarProxy 0.8.16 Released

PolarProxy 0.8.16 We are happy to announce a new release of the TLS decryption tool PolarProxy. The new version has been updated to support features like client certificates and a PCAP-over-IP connector.

Client Certificates

PolarProxy now supports client-authenticated TLS handshakes for outgoing connections to support sites that require mutual TLS (mTLS) authentication. The following example uses the PKCS#12 client certificate "client.p12" with password "pwd" to authenticate PolarProxy when connecting to "https://api.example.com":

./PolarProxy -p 10443,80,443 --clientcert api.example.com:client.p12:pwd

Thanks to Peter Lambrechtsen for the idea!

Bypassing Decryption for Specific Domains

There are situations when it isn't appropriate to decrypt the traffic passing through PolarProxy. The traffic might, for example, contain personal or confidential information. It might also not be possible to decrypt the traffic for technical reasons, such as when clients use certificate pinning or certificate transparency to validate the server certificate. We therefore recommend that such sites are put on a "bypass" list, i.e. a list of domains for which PolarProxy should let the encrypted traffic pass untouched to preserve the end-to-end encryption between the client and server.

PolarProxy's "--bypass <file>" option, which can be used to provide a regular expression list of domains not to decrypt, has now been acompanied by "--bypassexact <file>". The new --bypassexact option simply matches domains against the lines in <file> using string matching of the full domain name, no fancy-pants regex involved.

PCAP-over-IP Client

The new "--pcapoveripconnect" option can be used to let PolarProxy connect to a PCAP-over-IP listener and send it a live PCAP stream of decrypted traffic over TCP. This option complements PolarProxy's "--pcapoverip" option, which sets up a PCAP-over-IP listener that serves clients with the same PCAP stream. Thanks to Andy Wick for suggesting adding a PCAP-over-IP connector to PolarProxy!

The following command instructs PolarProxy to send a live PCAP stream with decrypted traffic to a local PCAP-over-IP listener:

./PolarProxy -p 10443,80,443 --pcapoveripconnect 127.0.0.1:57012

PolarProxy will automatically attempt to re-establish the PCAP-over-IP connection every 10 seconds if it goes down or cannot be established for some reason.

Only Store Packets When Instructed

PolarProxy no longer writes hourly rotated pcap files with decrypted packets to disk unless explicitly instructed to do so with "-o <directory>" or "-w <file>".

Flushing Buffered Packets to Disk

PolarProxy now periodically flushes buffered packets to disk every 60 seconds. The flush interval can be controlled with the "--autoflush <seconds>" option. The auto flush can also be disabled with "--autoflush 0".

No More Out-of-Quota Issues

We have also improved the quota handling for our privileged users, who have a license key that allows them to decrypt more than 10 GB or 10 000 TLS sessions per day. You should now be able to use your full daily quota without issues!

Posted by Erik Hjelmvik on Monday, 30 November 2020 07:45:00 (UTC/GMT)

Tags: #Netresec#PolarProxy#PCAP#TLS#PCAP-over-IP#pcapoverip#certificate

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=20Bf4b2


PolarProxy in Podman

PolarProxy + Podman Logo

Podman is a daemonless Linux container engine, which can be used as a more secure alternative to Docker. This blog post demonstrates how to run PolarProxy in a rootless container using Podman. If you still prefer to run PolarProxy in Docker, then please read our blog post "PolarProxy in Docker" instead.

Install Podman and fuse-overlayfs

Install Podman according to the official Podman installation instructions. Then install fuse-overlayfs, which is an overlay file system for rootless containers. Fuse-overlayfs can be installed in Debian/Ubuntu with "sudo apt install fuse-overlayfs" and in CentOS with "sudo yum install fuse-overlayfs".

Create a Podman Image for PolarProxy

Create a Dockerfile with the following contents:

FROM mcr.microsoft.com/dotnet/core/runtime:2.2
EXPOSE 10443
EXPOSE 10080
EXPOSE 57012
RUN groupadd -g 31337 polarproxy && useradd -m -u 31337 -g polarproxy polarproxy && mkdir -p /var/log/PolarProxy /opt/polarproxy && chown polarproxy:polarproxy /var/log/PolarProxy && curl -s https://www.netresec.com/?download=PolarProxy | tar -xzf - -C /opt/polarproxy
USER polarproxy
WORKDIR /opt/polarproxy/
ENTRYPOINT ["dotnet", "PolarProxy.dll"]
CMD ["-v", "-p", "10443,80,443", "-o", "/var/log/PolarProxy/", "--certhttp", "10080", "--pcapoverip", "57012"]

Save the Docker file as "Dockerfile" (no extension) in an empty directory and start a shell in that directory. Build a PolarProxy Podman image with:

podman build -f Dockerfile -t polarproxy

Test the PolarProxy Podman Image

Take the polarproxy Podman image for a test run. Start it with:

podman run -it --rm --name polarproxy -p 10443 localhost/polarproxy

Establish an HTTPS connection through PolarProxy by running this curl command from another shell on the same machine:

curl --insecure --connect-to www.netresec.com:443:127.0.0.1:10443 https://www.netresec.com/

If everything works alright, then curl should output HTML and the interactive Podman session running the polarproxy image should print something like:

<6>[10443] 127.0.0.1 -> N/A Connection from: 127.0.0.1:44122
<6>[10443] 127.0.0.1 -> www.netresec.com Connection request for: www.netresec.com from 127.0.0.1:44122
<6>[10443] 127.0.0.1 -> www.netresec.com Action: DECRYPT

Create a Podman Container for PolarProxy

Create directories "pcap" and "polarproxy", where PolarProxy should store the decrypted network traffic and its root CA certificate.

mkdir pcap polarproxy
podman unshare chown 31337:31337 pcap polarproxy

Create a container called "polarproxy", which has the "pcap" and "polarproxy" directories mounted as volumes. The service on TCP 10080 will serve the proxy's public root cert over HTTP. The localhost:57012 service is a Pcap-over-IP server, from which the decrypted network traffic can be streamed in real-time.

podman create --name polarproxy -v $(pwd)/pcap:/var/log/PolarProxy -v $(pwd)/polarproxy:/home/polarproxy -p 10443 -p 10080 -p 127.0.0.1:57012:57012 localhost/polarproxy

Create and enable a systemd user service that will run the container.

mkdir -p ~/.config/systemd/user/
podman generate systemd -n polarproxy > ~/.config/systemd/user/container-polarproxy.service
systemctl --user enable container-polarproxy.service

Start the systemd user service to activate the PolarProxy container.

systemctl --user start container-polarproxy.service

Verify that the service is running and that you can view the logs from PolarProxy.

systemctl --user status container-polarproxy.service
podman logs polarproxy

Expose PolarProxy to the Network

Create a firewall rule to redirect incoming TCP 443 packets to the PolarProxy service listening on port 10443.
sudo iptables -t nat -A PREROUTING -d 10.11.12.13 -p tcp --dport 443 -j REDIRECT --to 10443
Note: Replace "10.11.12.13" with the IP of the PolarProxy machine

Try making an HTTPS connection via PolarProxy from another PC on the network.

C:\> curl --insecure --resolve www.netresec.com:443:10.11.12.13 https://www.netresec.com/
Note: Replace "10.11.12.13" with the IP of the PolarProxy machine

Don't forget to save the firewall redirect rule if it is working as desired!

Redirect HTTPS and Trust the Root CA

You can now redirect outgoing TCP 443 traffic from your network to your Podman/PolarProxy host. Review the "Routing HTTPS Traffic to the Proxy" section on the PolarProxy page for recommendations on how to redirect outgoing traffic to PolarProxy.

Finally, configure the operating system, browsers and other applications that will get their TLS traffic proxied by PolarProxy to trust the root CA of the PolarProxy service running in your Podman container. Follow the steps in the "Trusting the PolarProxy root CA" section of the PolarProxy documentation in order to install the root cert.

Accessing Decrypted TLS Traffic

You should be able to access PCAP files with the decrypted HTTPS traffic in the "pcap" directory.

It is also possible view the decrypted traffic in real-time by using netcat and tcpdump as a Pcap-over-IP client like this:

nc localhost 57012 | tcpdump -nr - -X

It probably makes more sense to forward the decrypted traffic to an IDS or other type of network security monitoring tool though. See our blog posts "Sniffing Decrypted TLS Traffic with Security Onion" and "Capturing Decrypted TLS Traffic with Arkime" for instructions on how to forward the decrypted network traffic to a network monitoring solution like Security Onion or Arkime.

PolarProxy in Podman on ARM Linux

PolarProxy can also run on ARM Linux installations, such as a Raspberry Pi. However, the Dockerfile must be modified slightly in order to do so.

ARM 32-bit / AArch32 / ARMv7 If you're running an "arm32" Linux OS, then change the download link in the "RUN" instruction to the following URL:
https://www.netresec.com/?download=PolarProxy_linux-arm

ARM 64-bit / AArch64 / ARMv8 If you're running an "arm64" Linux OS, then change the download link in the "RUN" instruction to the following URL:
https://www.netresec.com/?download=PolarProxy_linux-arm64

Don't know if you're running a 32-bit or 64-bit OS? Run "uname -m" and check if the output says "armv7*" (arm32) or "armv8*" (arm64).

See our blog post "Raspberry PI WiFi Access Point with TLS Inspection" for more details about deploying PolarProxy on a Raspberry Pi.

ʕ•ᴥ•ʔ + 🦭 = 💜

Posted by Erik Hjelmvik on Tuesday, 27 October 2020 18:33:00 (UTC/GMT)

Tags: #PolarProxy#Docker#TLS#HTTPS#Proxy#curl#PCAP#Dockerfile#DNAT#container#arm32#arm64#AArch64#PCAP-over-IP#pcapoverip#systemctl#systemd

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=20A16ef


PolarProxy in Docker

PolarProxy + Docker

Our transparent TLS proxy PolarProxy is gaining lots of popularity due to how effective it is at generating decrypted PCAP files in combination with how easy it is to deploy. In this blog post we will show how to run PolarProxy in Docker.

Installation Instructions

Create a Dockerfile with the following contents:

FROM mcr.microsoft.com/dotnet/core/runtime:2.2
EXPOSE 10443
EXPOSE 10080
EXPOSE 57012
RUN groupadd -g 31337 polarproxy && useradd -m -u 31337 -g polarproxy polarproxy && mkdir -p /var/log/PolarProxy /opt/polarproxy && chown polarproxy:polarproxy /var/log/PolarProxy && curl -s https://www.netresec.com/?download=PolarProxy | tar -xzf - -C /opt/polarproxy
VOLUME ["/var/log/PolarProxy/", "/home/polarproxy/"]
USER polarproxy
WORKDIR /opt/polarproxy/
ENTRYPOINT ["dotnet", "PolarProxy.dll"]
CMD ["-v", "-p", "10443,80,443", "-o", "/var/log/PolarProxy/", "--certhttp", "10080", "--pcapoverip", "0.0.0.0:57012"]

Save the Docker file as "Dockerfile" (no extension) in an empty directory and start a shell in that directory with root privileges. Build the PolarProxy Docker image with:

docker build -t polarproxy-image .

Next, create a Docker container named "polarproxy":

docker create -p 443:10443 -p 10443:10443 -p 10080:10080 --name polarproxy polarproxy-image
The "-p" switches in this command define three DNAT rules that will get activated when the polarproxy container is started. The first DNAT rule forwards incoming TCP port 443 traffic to the polarproxy Docker container's transparent TLS proxy service on TCP port 10443. The second one does the same thing, but for incoming traffic to TCP 10443. The last one forwards TCP port 10080 traffic to a web server that delivers the public X.509 certificate of the proxy.

It is now time to start the polarproxy container:

docker start polarproxy

Verify that PolarProxy is running:

docker ps
docker logs polarproxy

Try fetching PolarProxy's public root CA certificate with curl and then connect to a website over HTTPS through the proxy:

curl -sL http://localhost:10080 | openssl x509 -inform DER -issuer -noout -dates
curl --insecure --connect-to www.netresec.com:443:127.0.0.1:10443 https://www.netresec.com/
curl --insecure --resolve www.netresec.com:443:127.0.0.1 https://www.netresec.com/

Redirect HTTPS and Trust the Root CA

You can now redirect outgoing TCP 443 traffic from your network to your Docker host. Review the "Routing HTTPS Traffic to the Proxy" section on the PolarProxy page for recommendations on how to redirect outgoing traffic to PolarProxy.

Finally, configure the operating system, browsers and other applications that will get their TLS traffic proxied by PolarProxy to trust the root CA of the PolarProxy service running in your Docker container. Follow the steps in the "Trusting the PolarProxy root CA" section of the PolarProxy documentation in order to install the root cert.

Docker Volumes

The Docker file we used in this blog post defines two volumes. The first volume is mounted on "/var/log/PolarProxy" in the container, which is where the decrypted network traffic will be stored as hourly rotated PCAP files. The second volume is the polarproxy home directory, under which PolarProxy will store its private root CA certificate.

The volumes are typically located under "/var/lib/docker/volumes" on the Docker host's file system. You can find the exact path by running:

docker volume ls
docker volume inspect <VOLUME_NAME>

Or use find to list *.pcap files in the Docker volumes directory:

find /var/lib/docker/volumes/ -name *.pcap
/var/lib/docker/volumes/7ebb3f56fd4ceab96[...]/_data/​proxy-201006-095937.pcap/var/lib/docker/volumes/7ebb3f56fd4ceab96[...]/_data/​proxy-201006-105937.pcap/var/lib/docker/volumes/7ebb3f56fd4ceab96[...]/_data/​proxy-201006-115937.pcap

The full path of your private PolarProxy Root CA certificate, which is located under "/home/polarproxy/" in the Docker container, can also be located using find:

find /var/lib/docker/volumes/ -name *.p12
/var/lib/docker/volumes/dcabbbac10e1b1461[...]/_data/​.local/share/PolarProxy/​e249f9c497d7b5c41339f153a31eda1c.p12

We recommend reusing the "/home/polarproxy/" volume, when deploying new PolarProxy instances or upgrading to a new version of PolarProxy, in order to avoid having to re-configure clients to trust a new root CA every time a new PolarProxy container is created.

PolarProxy in Docker on ARM Linux

PolarProxy can also run on ARM Linux installations, such as a Raspberry Pi. However, the Dockerfile must be modified slightly in order to do so.

ARM 32-bit / AArch32 / ARMv7 If you're running an "arm32" Linux OS, then change the download link in the "RUN" instruction to the following URL:
https://www.netresec.com/?download=PolarProxy_linux-arm

ARM 64-bit / AArch64 / ARMv8 If you're running an "arm64" Linux OS, then change the download link in the "RUN" instruction to the following URL:
https://www.netresec.com/?download=PolarProxy_linux-arm64

Don't know if you're running a 32-bit or 64-bit OS? Run "uname -m" and check if the output says "armv7*" (arm32) or "armv8*" (arm64).

See our blog post "Raspberry PI WiFi Access Point with TLS Inspection" for more details about deploying PolarProxy on a Raspberry Pi (without Docker).

Credits

We'd like to thank Jonas Lejon for contacting us back in February about the work he had done to get PolarProxy running in Docker. We used Jonas' work as a starting point when building the installation instructions in this how-to guide.

We also want to thank Erik Ahlström for providing valuable feedback on the instructions in this guide.

ʕ•ᴥ•ʔ + 🐳 = 💜

Posted by Erik Hjelmvik on Wednesday, 07 October 2020 08:09:00 (UTC/GMT)

Tags: #PolarProxy#Docker#TLS#HTTPS#Proxy#TLSI#Dockerfile#curl#x509#X.509#PCAP#DNAT#container#DNAT#arm32#arm64#AArch64#PCAP-over-IP#pcapoverip

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=20Accbd


Discovered Artifacts in Decrypted HTTPS

We released a PCAP file earlier this year, which was recorded as part of a live TLS decryption demo at the CS3Sthlm conference. The demo setup used PolarProxy running on a Raspberry Pi in order to decrypt all HTTPS traffic and save it in a PCAP file as unencrypted HTTP.

Laptop, Raspberry Pi, PolarProxy, Internet ASCII

This capture file was later used as a challenge for our twitter followers, when we made the following announcement:

PCAP CHALLENGE!
The capture file released in this blog post contains a few interesting things that were captured unintentionally. Can you find anything strange, funny or unexpected in the pcap file? (1/2)

Followed by this message:

The person to submit the most interesting answer wins a “PCAP or it didn’t happen” t-shirt. Compete by including your discovery in a retweet or reply to this tweet, or in an email to info(at)netresec.com. We want your answers before the end of January. (2/2)

We'd like to thank everyone who submitted answers in this challenge, such as David Ledbetter, Christoffer Strömblad, RunΞ and Chris Sistrunk.

We're happy to announce that the winner of our challenge is David Ledbetter. Congratulations David!

So what were the interesting thing that could be found in the released capture file? Below is a short summary of some things that can be found.

Telemetry data sent to mozilla.org

A surprising amount of information about the Firefox browser was sent to incoming.telemetry.mozilla.org, including things like:

  • Active browser addons
  • Active browser plugins
  • Firefox profile creation date
  • Browser search region
  • Default search engine
  • Regional locales
  • Screen width
  • Screen height
  • CPU vendor, family and model
  • HDD model, revision and type
  • Installed RAM
  • Operating system
  • Etc..

Here's an excerpt showing a part of the data sent to Mozilla:

"build": { "applicationId": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", "applicationName": "Firefox", "architecture": "x86-64", "buildId": "20191002194346", "version": "69.0.2", "vendor": "Mozilla", "displayVersion": "69.0.2", "platformVersion": "69.0.2", "xpcomAbi": "x86_64-gcc3", "updaterAvailable": false }, "partner": { "distributionId": "canonical", "distributionVersion": "1.0", "partnerId": "ubuntu", "distributor": "canonical", "distributorChannel": "ubuntu", "partnerNames": [ "ubuntu" ] }, "system": { "memoryMB": 3943, "virtualMaxMB": null, "cpu": { "count": 1, "cores": 1, "vendor": "GenuineIntel", "family": 6, "model": 42, "stepping": 7, "l2cacheKB": 256, "l3cacheKB": 4096, "speedMHz": null, "extensions": [ "hasMMX", "hasSSE", "hasSSE2", "hasSSE3", "hasSSSE3", "hasSSE4_1", "hasSSE4_2", "hasAVX", "hasAES" ] }, "os": { "name": "Linux", "version": "5.0.0-31-generic", "locale": "en-US" }, "hdd": { "profile": { "model": null, "revision": null, "type": null }, "binary": { "model": null, "revision": null, "type": null }, "system": { "model": null, "revision": null, "type": null } }, "gfx": { "D2DEnabled": null, "DWriteEnabled": null, "ContentBackend": "Skia", "Headless": false, "adapters": [ { "description": "llvmpipe (LLVM 8.0, 256 bits)", "vendorID": "0xffff", "deviceID": "0xffff", "subsysID": null, "RAM": 3942, "driver": null, "driverVendor": "mesa/llvmpipe", "driverVersion": "19.0.8.0", "driverDate": null, "GPUActive": true } ], "monitors": [ { "screenWidth": 681, "screenHeight": 654 } ], "features": { "compositor": "basic", "gpuProcess": { "status": "unavailable" }, "wrQualified": { "status": "blocked-vendor-unsupported" }, "webrender": { "status": "opt-in" } } }, "appleModelId": null }, "settings": { "blocklistEnabled": true, "e10sEnabled": true, "e10sMultiProcesses": 8, "telemetryEnabled": false, "locale": "en-US", "intl": { "requestedLocales": [ "en-US" ], "availableLocales": [ "en-US", "en-CA", "en-GB" ], "appLocales": [ "en-US", "en-CA", "en-GB", "und" ], "systemLocales": [ "en-US" ], "regionalPrefsLocales": [ "sv-SE" ], "acceptLanguages": [ "en-US", "en" ] }, "update": { "channel": "release", "enabled": true, "autoDownload": false }, "userPrefs": { "browser.cache.disk.capacity": 1048576, "browser.search.region": "SE", "browser.search.widget.inNavBar": false, "network.trr.mode": 2 }, "sandbox": { "effectiveContentProcessLevel": 4 }, "addonCompatibilityCheckEnabled": true, "isDefaultBrowser": false, "defaultSearchEngine": "google", "defaultSearchEngineData": { "name": "Google", "loadPath": "[distribution]/searchplugins/locale/en-US/google.xml", "origin": "default", "submissionURL": "https://www.google.com/search?client=ubuntu&channel=fs&q=&ie=utf-8&oe=utf-8" } }, "profile": { "creationDate": 18183, "firstUseDate": 18183 }

You can use the following Wireshark display filter to find all the data sent to Mozilla:

http.request.method eq POST and http.host contains telemetry

Public IP Revealed in PCAP

The client's IP address was 192.168.4.20, which is part of the RFC 1918 192.168/16 private address space. It's therefore safe to assume that the client was behind a NAT (the client was in fact behind a double NAT). However, we noticed that the public IP of the client was revealed through multiple services in the captured network traffic. One of these services is the advertising exchange company AppNexus (adnxs.com), which sent the client's public IP address 193.235.19.252 in an X-Proxy-Origin HTTP header.

X-Proxy-Origin HTTP header in Wireshark

You can use the following Wireshark/tshark display filter to find X-Proxy-Origin headers:

http.response.line matches "x-proxy-origin" or http2.header.name matches "x-proxy-origin"

We are using the "matches" operator here instead of "contains" or "==" because we want to perform case insensitive matching. You might also notice that we need a completely different display filter syntax to match HTTP/2 headers compared to what we are used to with HTTP/1.1.

Monty Python "Majestik møøse" reference in reddit x-header

The reddit server 151.101.85.140 sends an HTTP/2 header called "x-moose" with a value of "majestic".

x-moose 1 : majestic header from reddit

This header refers to the opening credits of Monty Python and the Holy Grail.

Wi nøt trei a høliday in Sweden this yër?

Posted by Erik Hjelmvik on Tuesday, 17 March 2020 09:00:00 (UTC/GMT)

Tags: #HTTP/2#http2#TLS#decrypt#TLSI#PolarProxy#NetworkMiner#Wireshark#CS3Sthlm#CS3#Forensics#PCAP#ASCII-art

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=2030e17


Reverse Proxy and TLS Termination

PolarProxy is primarily a TLS forward proxy, but it can also be used as a TLS termination proxy or reverse TLS proxy to intercept and decrypt incoming TLS traffic, such as HTTPS or IMAPS, before it is forwarded to a server. The proxied traffic can be accessed in decrypted form as a PCAP formatted data stream, which allows real-time analysis of the decrypted traffic by an IDS as well as post incident forensics with Wireshark.

PolarProxy version 0.8.15 and later can import an existing X.509 server certificate (aka leaf certificate or end-entity certificate) in order to perform the TLS decryption using a valid certificate signed by a trusted certificate authority. If no server certificate is provided, then PolarProxy falls back to generating server certificates on the fly and signing them with its own root CA certificate.

There are two principal ways to run PolarProxy as a reverse proxy, either as a TLS termination proxy or as a reverse proxy that decrypts and re-encrypts the traffic.

PolarProxy as a TLS Termination Proxy

TLS Termination Proxy

The TLS termination proxy mode is useful in order to offload the task of performing TLS encryption to PolarProxy instead of doing the decryption on the web server. This mode can also be used when the proxied services don’t support TLS encryption, such as legacy web servers or servers hosting other unencrypted services that you want to secure with TLS.

The following command sequence shows how to create a Let’s Encrypt SSL certificate, convert it to the PKCS#12 format, and load the server certificate into PolarProxy to terminate incoming HTTPS connections. In this setup PolarProxy decrypts the TLS traffic and relays the HTTP traffic to the web server on TCP port 80.

sudo certbot certonly --manual --preferred-challenges dns -d example.com,www.example.com

sudo openssl pkcs12 -export -out /etc/example.p12 -inkey /etc/letsencrypt/live/example.com/privkey.pem -in /etc/letsencrypt/live/example.com/fullchain.pem --passout pass:PASSWORD

sudo mkdir /var/log/TlsTerminationProxy/

sudo ./PolarProxy --terminate --connect 10.1.2.3 --nosni www.example.com --servercert example.com,www.example.com:/etc/example.p12:PASSWORD -p 443,80,80 -o /var/log/TlsTerminationProxy/

Here’s a breakdown of the arguments sent to PolarProxy:

  • --terminate : Terminate incoming TLS sessions and forward proxied traffic in unencrypted form.
  • --connect 10.1.2.3 : Forward all proxied traffic to 10.1.2.3 instead of connecting to the host name provided in the SNI extension of the TLS ClientHello message.
  • --nosni www.example.com : Treat incoming TLS sessions that don’t define a host name with the SNI extension as if they wanna to connect to “www.example.com”.
  • --servercert example.com,www.example.com:/etc/example.p12:PASSWORD : Use the server certificate “/etc/example.p12” for incoming connections to “example.com” and “www.example.com”.
  • -p 443,80,80 : Listen on TCP port 443, save decrypted traffic in PCAP file as if it was directed to port 80, forward decrypted traffic to port 80.
  • -o /var/log/TlsTerminationProxy/ : Save decrypted traffic to hourly rotated PCAP files in “/var/log/TlsTerminationProxy/”.

PolarProxy is a generic TLS proxy that doesn’t care what application layer protocol the TLS tunnel carries. So if you want to terminate the TLS encryption of incoming IMAPS sessions as well, then simply append an additional argument saying “-p 993,143,143” to also forward decrypted IMAP sessions to 10.1.2.3. This method can be used in order to wrap almost any TCP based protocol in a TLS tunnel, which can be useful for privacy reasons as well as to prevent network monitoring tools from detecting the actual application layer protocol.

PolarProxy as a Reverse TLS Proxy

Reverse TLS Proxy

There are setups for which it is preferable to also encrypt the internal sessions between PolarProxy and the final server. One such setup is when the server is hosting a web service with support for the HTTP/2 protocol, which in practice always uses TLS. Luckily PolarProxy is designed to decrypt and re-encrypt proxied traffic while also forwarding important TLS parameters, such as ALPN and SNI, between the internal and external TLS sessions.

To use TLS encryption on the inside as well as outside of PolarProxy, simply do as explained in the previous TLS termination section, but remove the “--terminate” argument and change the port argument to “-p 443,80,443” like this:

sudo ./PolarProxy --connect 10.1.2.3 --nosni www.example.com --servercert example.com,www.example.com:/etc/example.p12:PASSWORD -p 443,80,443 -o /var/log/ReverseTlsProxy/

PolarProxy will save the decrypted traffic as cleartext HTTP (or HTTP/2) to PCAP files in the “/var/log/ReverseTlsProxy/” directory.

Real-Time Analysis of Decrypted Traffic

Both the external (client-to-proxy) and internal (proxy-to-server) TCP sessions, in the reverse TLS proxy example above, are encrypted with TLS. This prevents passive network security monitoring tools, such as IDSs, DPI and DLP appliances, from analyzing the application layer data being sent and received. The PCAP files written to “/var/log/ReverseTlsProxy/” can be a valuable forensic asset when investigating an incident, but a real-time stream of the decrypted data is needed in order to swiftly detect and alert on potential security breaches and other incidents.

PolarProxy’s “--pcapoverip” option can be used to provide such a real-time stream of the decrypted data passing through the proxy. This data can easily be sent to a network interface using tcpreplay, as explained in our blog post “Sniffing Decrypted TLS Traffic with Security Onion”.

Security Considerations

The examples shown in this blog post all run PolarProxy with root privileges using sudo, which can be dangerous from a security perspective. PolarProxy is actually designed to be run without root privileges, but doing so prevents it from listening on a port below 1024. Luckily, this issue can easily be overcome with a simple port forwarding or redirect rule. The following iptables redirect rule can be used if PolarProxy is listening on TCP port 20443 and incoming HTTPS request are arriving to the eth0 interface of the proxy:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to 20443

PolarProxy does not support loading settings from a config file. The password for the PKCS12 certificate will therefore need to be supplied on the command line, which can make it visible from a process listing. If this is a concern for you, then please consider using “hidepid” to hide processes from other users. You can find instructions on how to use hidepid in hardening guides for Debian, Arch, SUSE and most other Linux flavors.

Posted by Erik Hjelmvik on Thursday, 12 March 2020 15:45:00 (UTC/GMT)

Tags: #PolarProxy#TLS#SSL#PCAP#decrypt#HTTPS#HTTP#HTTP/2#http2#IMAPS#decrypt#ASCII-art

Share: Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=2033d38

2020 January

Sniffing Decrypted TLS Traffic with Security Onion

Sharing a PCAP with Decrypted HTTPS

2019 December

Installing a Fake Internet with INetSim and PolarProxy

2019 November

The NSA HSTS Security Feature Mystery

2019 September

Raspberry PI WiFi Access Point with TLS Inspection

2019 June

PolarProxy Released

2018 February

Examining an x509 Covert Channel

2017 September

Hunting AdwindRAT with SSL Heuristics

2014 October

Verifying Chinese MITM of Yahoo

2014 September

Analysis of Chinese MITM on Google

2013 April

Detecting TOR Communication in Network Traffic

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec