NETRESEC Network Security Blog - Tag : Pcap-over-IP


Sniffing Decrypted TLS Traffic with Security Onion

Wouldn't it be awesome to have a NIDS like Snort, Suricata or Zeek inspect HTTP requests leaving your network inside TLS encrypted HTTPS traffic? Yeah, we think so too! We have therefore created this guide on how to configure Security Onion to sniff decrypted TLS traffic with help of PolarProxy.

Network drawing with Clients, SecurityOnion and the Internet

PolarProxy is a forward TLS proxy that decrypts incoming TLS traffic from clients, re-encrypts it and forwards it to the server. One of the key features in PolarProxy is the ability to export the proxied traffic in decrypted form using the PCAP format (a.k.a. libpcap/tcpdump format). This makes it possible to read the decrypted traffic with external tools, without having to perform the decryption again. It also enables packet analysis using tools that don't have built-in TLS decryption support.

This guide outlines how to configure PolarProxy to intercept HTTPS traffic and send the decrypted HTTP traffic to an internal network interface, where it can be sniffed by an IDS.

STEP 1 ☆ Install Ubuntu

Download and install the latest SecurityOnion ISO image, but don't run the "Setup" just yet.

STEP 2 ☆ Add a Dummy Network Interface

Add a dummy network interface called "decrypted", to which decrypted packets will be sent.

ip link add decrypted type dummy
ip link set decrypted arp off up
Add the commands above to /etc/rc.local before "exit 0" to have the network interface automatically configured after reboots.

dummy interface in rc.local

STEP 3 ☆ Install Updates

Install updates in Security Onion by running "sudo soup".

STEP 4 ☆ Run the Security Onion Setup

Run the Security Onion setup utility by double-clicking the "Setup" desktop shortcut or executing "sudo sosetup" from a terminal. Follow the setup steps in the Production Deployment documentation and select "decrypted" as your sniffing interface.

Sniffing Interface Selection Window

Reboot and run Setup again to continue with the second phase of Security Onion's setup. Again, select "decrypted" as the interface to be monitored.

STEP 5 ☆ Install PolarProxy Service

Download and install PolarProxy:

sudo adduser --system --shell /bin/bash proxyuser
sudo mkdir /var/log/PolarProxy
sudo chown proxyuser:root /var/log/PolarProxy/
sudo chmod 0775 /var/log/PolarProxy/

sudo su - proxyuser
mkdir ~/PolarProxy
cd ~/PolarProxy/
curl https://www.netresec.com/?download=PolarProxy | tar -xzf -
exit

sudo cp /home/proxyuser/PolarProxy/PolarProxy.service /etc/systemd/system/PolarProxy.service

Edit /etc/systemd/system/PolarProxy.service and add "--pcapoverip 57012" at the end of the ExecStart command.

--pcapoverip 57012 in PolarProxy.service

Start the PolarProxy systemd service:

sudo systemctl enable PolarProxy.service
sudo systemctl start PolarProxy.service

STEP 6 ☆ Install Tcpreplay Service

The decrypted traffic can now be accessed via PolarProxy's PCAP-over-IP service on TCP 57012. We can leverage tcpreplay and netcat to replay these packets to our dummy network interface in order to have them picked up by Security Onion.

nc localhost 57012 | tcpreplay -i decrypted -t -
However, it's better to create a systemd service that does this automatically on bootup. We therefore create a file called /etc/systemd/system/tcpreplay.service with the following contents:
[Unit]
Description=Tcpreplay of decrypted traffic from PolarProxy
After=PolarProxy.service

[Service]
Type=simple
ExecStart=/bin/sh -c 'nc localhost 57012 | tcpreplay -i decrypted -t -'
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target

Start the tcpreplay systemd service:

sudo systemctl enable tcpreplay.service
sudo systemctl start tcpreplay.service

STEP 7 ☆ Add firewall rules

Security Onion only accepts incoming connections on TCP 22 by default, we also need to allow connections to TCP port 10443 (proxy port), and 10080 (root CA certificate download web server). Add allow rules for these services to the Security Onion machine's firewall:

sudo ufw allow in 10443/tcp
sudo ufw allow in 10080/tcp

Verify that the proxy is working by running this curl command on a PC connected to the same network as the Security Onion machine:

curl --insecure --connect-to www.netresec.com:443:[SecurityOnionIP]:10443 https://www.netresec.com/
Note: You can even perform this test from a Win10 PC, since curl is included with Windows 10 version 1803 and later.

Add the following lines at the top of /etc/ufw/before.rules (before the *filter section) to redirect incoming packets on TCP 443 to PolarProxy on port 10443.

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i enp0s3 -p tcp --dport 443 -j REDIRECT --to 10443
COMMIT

Note: Replace "enp0s3" with the Security Onion interface to which clients will connect.

After saving before.rules, reload ufw to activate the port redirection:

sudo ufw reload

Verify that you can reach the proxy on TCP 443 with this command:

curl --insecure --resolve www.netresec.com:443:[SecurityOnionIP] https://www.netresec.com/

STEP 8 ☆ Redirect HTTPS traffic to PolarProxy

It's now time to configure a client to run its HTTPS traffic through PolarProxy. Download and install the PolarProxy X.509 root CA certificate from PolarProxy's web service on TCP port 10080:

http://[SecurityOnionIP]:10080/polarproxy.cer

Install the certificate in the operating system and browser, as instructed in the PolarProxy documentation.

You also need to forward packets from the client machine to the Security Onion machine running PolarProxy. This can be done either by configuring a local NAT rule on each monitored client (STEP 8.a) or by configuring the default gateway's firewall to forward HTTPS traffic from all clients to the proxy (STEP 8.b).

STEP 8.a ☆ Local NAT

Use this firewall rule on a Linux client to configure it to forward outgoing HTTPS traffic to the Security Onion machine:

sudo iptables -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to [SecurityOnionIP]

STEP 8.b ☆ Global NAT Network drawing Firewall, PolarProxy, Clients

If the client isn't running Linux, or if you wanna forward HTTPS traffic from a whole network to the proxy, then apply the following iptables rules to the firewall in front of the client network. See "Routing Option #2" in the PolarProxy documentation for more details.

  1. Add a forward rule on the gateway to allow forwarding traffic to our PolarProxy server:
    sudo iptables -A FORWARD -i eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -m state --state NEW -j ACCEPT
  2. Add a DNAT rule to forward 443 traffic to PolarProxy on port 10443:
    sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j DNAT --to [SecurityOnionIP]:10443
  3. If the reverse traffic from PolarProxy to the client doesn't pass the firewall (i.e. they are on the same LAN), then we must add this hide-nat rule to fool PolarProxy that we are coming from the firewall:
    sudo iptables -t nat -A POSTROUTING -o eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -j MASQUERADE
For other network configurations, please see the various routing setups in the PolarProxy documentation.

STEP 9 ☆ Inspect traffic in SecurityOnion

Wait for the Elastic stack to initialize, so that the intercepted network traffic becomes available through the Kibana GUI. You can check the status of the elastic initialization with "sudo so-elastic-status".

You should now be able to inspect decrypted traffic in Security Onion using Kibana, Squert, Sguil etc., just as if it was unencrypted HTTP.

Bro HTTP traffic in Kibana Image: Kibana showing HTTP traffic info from decrypted HTTPS sessions

MIME types in Kibana Image: MIME types in Kibana

NIDS alerts in Kibana Image: NIDS alerts from payload in decrypted traffic shown in Kibana

Snort alerts in Squert Image: Snort alerts from decrypted traffic shown in Squert

Security Considerations and Hardening

Security Onion nodes are normally configured to only allow access by SOC/CERT/CSIRT analysts, but the setup described in this blog post requires that "normal" users on the client network can access the PolarProxy service running on the Security Onion node. We therefore recommend installing PolarProxy on a dedicated Security Onion Forward Node, which is configured to only monitor traffic from the proxy.

We also recommend segmenting the client network from the analyst network, for example by using separate network interfaces on the Security Onion machine or putting it in a DMZ. Only the PolarProxy service (TCP 10080 and 10443) should be accessable from the client network.

PolarProxy could be used to pivot from the client network into the analyst network or to access the Apache webserver running on the Security Onion node. For example, the following curl command can be used to access the local Apache server running on the Security Onion machine via PolarProxy:

curl --insecure --connect-to localhost:443:[SecurityOnionIP]:10443 https://localhost/
We therefore recommend adding firewall rules that prevent PolarProxy from accessing the analyst network as well as the local Apache server.

Hardening Steps:

  • Configure the Security Onion node as a Forward Node
  • Segment client network from analyst network
  • Add firewall rules to prevent PolarProxy from accessing services on the local machine and analyst network

For additional info on hardening, please see the recommendations provided by Wes Lambert on the Security-Onion mailing list.

Posted by Erik Hjelmvik on Monday, 20 January 2020 09:40:00 (UTC/GMT)

Tags: #SecurityOnion #Security Onion #PCAP #Bro #PolarProxy #Snort #Suricata #TLS #SSL #HTTPS #tcpreplay #PCAP-over-IP #IDS #NIDS #netcat #curl

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=2013af4


Running NetworkMiner on Mac OS X

Apple Logo

The following is a guest blog post written by Jonas Lejon from the Swedish IT security company Triop, which specialize in crypto, reverse engineering and penetration testing.

This guide describes how to get NetworkMiner running on Mac OS X Mavericks (version 10.9.3).

First of all, download NetworkMiner and then go to the Mono downloads page and get the latest version for Mac OS X


After the download of “Mono MRE installer” has completed, just run the installer:


Mono Framework Installer

Press “Continue” to proceed installing the Mono Framework using the guided installer.

When the Mono Framework has been installed you can extract the downloaded NetworkMiner zip archive. Then start NetworkMiner from the terminal like this:

$ mono --arch=32 NetworkMiner.exe
NetworkMiner 1.6 on Mac OS X - Click To Enlarge
 

Live sniffing with NetworkMiner on Mac OS X

Live sniffing with WinPcap or Raw Sockets is only available when running NetworkMiner in Windows. However, live sniffing can still be achieved on Mac OSX (as well as in Linux) by using the PCAP-over-IP functionality. Simply select [File > Receive PCAP over IP] or press [Ctrl]+R and select a TCP port to listen on (TCP 57012 is the default port).

Pcap-over-IP in NetworkMiner 1.6 on Mac OS X

Press the “Start Receiving” button and then use tcpdump to do live sniffing and forward all captured packets to NetworkMiner like this:

$ sudo tcpdump -i en0 -s0 -U -w - | nc localhost 57012

The preferred way to use NetworkMiner is, however, to load previously captured packets in a PCAP file and let NetworkMiner dig out all interesting details like transmitted files, images, messages, SSL certificates etc.

For more info on how to run NetworkMiner on other operating systems, please see our previous blog posts HowTo install NetworkMiner in Ubuntu Fedora and Arch Linux and No more Wine - NetworkMiner in Linux with Mono.


 

UPDATE

Microsoft .NET Windows.Forms GUI applications don't run on 64 bit macOS systems running Mono. This will cause the application to hang/freeze during startup when the GUI window is about to be rendered, throwing errors such as:

  • Unable to start NetworkMiner: An exception was thrown by the type initializer for System.Windows.Forms.WindowsFormsSynchronizationContext
  • Unhandled Exception: System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeEngine
Fortunately Mono can be configured to run using a 32-bit architecture with the --arch=32 argument like this:

$ mono --arch=32 /opt/NetworkMiner/NetworkMiner.exe

We'd like to thank Fredrik Pettai for reporting this issue and Joel Langill for suggesting the workaround.


 

UPDATE 190627

You can also use homebrew to install mono on macOS like this:

brew update && brew install mono

Posted by Jonas Lejon on Tuesday, 24 June 2014 21:25:00 (UTC/GMT)

Tags: #Mac #macOS #NetworkMiner #Mono #tcpdump #PCAP-over-IP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=146F525


NetworkMiner 1.6 Released

We've released version 1.6 of NetworkMiner today!

Confetti in Toronto by Winnie Surya Image credits: Confetti in Toronto by Winnie Surya

The new features in NetworkMiner 1.6 include:

  • Drag-and-Drop
    Reassembled files and images can be opened with external tools by drag-and-dropping items from NetworkMiner's Files or Images tabs onto your favorite editor or viewer.

  • Email extraction
    Improved extraction of emails and attachments sent over SMTP.

  • DNS analysis
    Failed DNS lookups that result in NXDOMAIN and SERVFAIL are displayed in the DNS tab along with the flags in the DNS response.

  • Live sniffing
    Improved live sniffing performance.

  • PCAP-over-IP
    Remote live sniffing enabled by bringing the PCAP-over-IP feature into the free open source version of NetworkMiner.


Identifying Malware DNS lookups

NetworkMiner Professional 1.6 with DNS traffic from the Contagio Kuluoz-Asprox

DNS traffic from the Kuluoz-Asprox botnet (PCAP file available via Contagio)

Note the NXDOMAIN responses and “No” in Alexa top 1 million column in the screenshot above; these domains are probably generated by a domain generation algorithm (DGA).

Live Sniffing with Pcap-over-IP

The PCAP-over-IP functionality enables live sniffing also on non-Windows machines, simply by running tcpdump (or dumpcap) and netcat like this:

# tcpdump -i eth0 -s0 -U -w - | nc localhost 57012
For more information about how to run NetworkMiner in Linux, please read our HowTo install NetworkMiner in Ubuntu Fedora and Arch Linux blog post.

To receive the Pcap-over-IP stream in NetworkMiner, simply press Ctrl+R and select a TCP port.

NetworkMiner Pcap-over-IP

For more information about this feature please see our previous blog post about the PCAP‑over‑IP feature.

NetworkMiner Professional

The professional version of NetworkMiner additionally contains the following improvements of the command line tool NetworkMinerCLI:

  • Enabled reading of PCAP and PcapNG data from standard input (STDIN)
  • Full support for PCAP-over-IP
  • More detailed DNS logging in NetworkMinerCLI's CSV export of DNS responses

The ability to read PCAP data from STDIN with NetworkMinerCLI makes it really simple to do live extraction of emails and email attachments. Here's an example showing how to do live SMTP extraction in Linux:

# tcpdump -i eth0 -s0 -w - port 25 or 587 | mono NetworkMinerCLI.exe -r - -w /var/log/smtp_extraction/

The syntax for extracting emails and attachments in Windows is very similar:

C:\>dumpcap.exe -i 1 -f "port 25 or 587" -w - | NetworkMinerCLI.exe -r -

The TCP ports 25 and 587, which are used in the capture filter above, are the standard port numbers for SMTP. In order to do live extraction of files sent over HTTP, simply use “port 80” as capture filter instead. Likewise, X.509 certificates can also be extracted from HTTPS sessions simply by using “port 443” as capture filter.

Download NetworkMiner 1.6

The most recent release of the free (open source) version of NetworkMiner can be downloaded from SourceForge or our NetworkMiner product page. Paying customers can download an update for NetworkMiner Professional from our customer portal.

Credits

We would like to thank Dan Eriksson (FM CERT) and Lenny Hansson (Danish GovCERT) for submitting bug reports and feature requests.

Posted by Erik Hjelmvik on Monday, 16 June 2014 11:00:00 (UTC/GMT)

Tags: #Netresec #NetworkMiner #Professional #SMTP #Extract #DNS #PCAP-over-IP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1463B56


NetworkMiner 1.1 Released

We are today very proud to release version 1.1 of NetworkMiner!

NetworkMiner Logo

The new releases of NetworkMiner (open source version) and NetworkMiner Professional (commercial version) includes the following features:

  • Extraction of parameters sent to Google Analytics into NetworkMiner's “Host Details”. These parameters include: screen resolution, color depth, browser language and flash version.
  • You can drag-and-drop one or multiple pcap files onto NetworkMiner.exe to have it start up and begin loading the dropped pcap files. You can also submit your pcap files as arguments from the command line.
  • Multiple SMB/CIFS and NetBIOS improvements, such as support for multiple simultaneous SMB file transfers over the same TCP session as well as support for NetBIOS Session Service keep-alive messages.
  • Added support for Point-to-Point Protocol (PPP) frames in pcap files.
  • Improved stability when loading pcap files. Thanks to psteier for identifying this bug.

NetworkMiner Professional additionally includes support for Pcap-over-IP, which comes in very handy when you need to access pcap files or network traffic from remote machines or devices. There is, however, no support for Pcap-over-IP in the open source version of NetworkMiner.

Upgrading from NetworkMiner Pro 1.0

We offer free upgrades for users of NetworkMiner Professional 1.0. Just send an email to info [at] netresec.com with your license number (which you can find under the menu “Help” > “About Network Miner”) and say that you'd like to upgrade to version 1.1.

Posted by Erik Hjelmvik on Thursday, 15 September 2011 17:25:00 (UTC/GMT)

Tags: #Netresec #NetworkMiner #Pcap-over-IP

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=119BFC1


Pcap-over-IP in NetworkMiner

Pcap over IP network protocol stack

Version 1.1 of NetworkMiner is soon to be released by us at Netresec. I would therefore like to give you a sneak preview of a simple yet very useful feature that we've added. We call this new feature “Pcap-over-IP”, which is a name originally coined by Packet Forensics.

With Pcap-over-IP you can have NetworkMiner read a pcap file (or libpcap formatted data in general) or over a TCP socket instead of getting it via the file system. The easiest way to send a pcap file over a TCP socket is to pipe a pcap file to netcat like this:

# cat sniffed.pcap | nc 192.168.1.20 57012

In this example I'd be running NetworkMiner on a PC with IP 192.168.1.20 and have Pcap‑over‑IP listening to TCP port 57012. NetworkMiner will save the received packets to disk as well as parse and display the contents of the packets in the GUI when receiving the Pcap‑over‑IP stream.

NetworkMiner receiving Pcap-over-IP data

Pcap-over-IP also allows me to do live network sniffing with dumpcap from my local Windows machine and pipe the captured packets to NetworkMiner via a TCP socket, using Netcat for Windows like this:

C:\Program Files\Wireshark>dumpcap -i 4 -P -w - | C:\Tools\Netcat\nc.exe 127.0.0.1 57012

Note that the “-w -” switch tells dumpcap to push the raw libpcap formated data to standard output (stdout) rather than saving it to a pcap file.

The reason for using dumcap to perform the live sniffing rather than using the built in packet capturing functionality of NetworkMiner is that dumpcap is an extremely reliably tool when it comes to capturing packets. So by sniffing with dumpcap instead of NetworkMiner you minimize the risk of dropping some packets.

I can also use Pcap-over-IP to capture network traffic from a remote PC or device. I can, for example use tcpdump to sniff traffic on the external interface of my Linux-based firewall and push it to an analyst station like this:

# tcpdump -i eth1 -s 0 -U -w - | nc 192.168.1.20 57012

I can also perform remote WiFi sniffing with dumpcap or tcpdump from a Linux machine and send the sniffed packets to NetworkMiner with netcat like this:

# iwconfig wlan0 mode monitor
# iwconfig wlan0 channel 4
# dumpcap -i wlan0 -P -w - | nc 192.168.1.20 57012

It is even possible to receive multiple PCAP streams simultaneously with NetworkMiner. This way I could have 14 dumpcap or tcpdump processes sniffing each individual IEEE 802.11 channel, while monitoring all the captured traffic in real-time with a single instance of NetworkMiner. However, note that this would require 14 sniffer computers or a single sniffer machine with 14 WiFi cards.

SSL encryption

Don't like sending your pcap files in cleartext over the network? That's just fine, we've also implemented support for SSL/TLS encryption in NetworkMiner. You can use the great multipurpose relay tool socat to read your pcap file and have it encrypted with SSL while transiting the network like this:

# socat GOPEN:sniffed.pcap SSL:192.168.1.20:57013,verify=0

You can also use socat when doing live sniffing like this:

# tcpdump -i br0 -s 0 -U -w - | socat - SSL:192.168.1.20:57013,verify=0

Warning: Always make sure you don't sniff your own Pcap-over-IP stream when sending packets to NetworkMiner. You will otherwise construct a feedback loop, which will fill up the tubes. If you need to sniff the same interface as you are using to perform the Pcap‑over‑IP transfer, then make sure to use BPF to filter out the port number used for your Pcap‑over‑IP transfer like this:

# tcpdump -i ppp0 -U -w - not port 57012 | nc 192.168.1.20 57012

UPDATE June 16, 2014

With the release of NetworkMiner 1.6 we've made the PCAP‑over‑IP functionality available in the free open source edition of NetworkMiner. We have also integrated PCAP‑over‑IP into NetworkMinerCLI, i.e. the command line version of NetworkMiner Professional.

Posted by Erik Hjelmvik on Wednesday, 07 September 2011 09:22:00 (UTC/GMT)

Tags: #Netresec #Pcap-over-IP #Pcap #netcat #tcpdump #dumpcap #TCP #SSL #TLS

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=119B126

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)