NETRESEC Network Security Blog - Tag : Suricata

rss Google News

What is PCAP over IP?

PCAP over IP

PCAP-over-IP is a method for reading a PCAP stream, which contains captured network traffic, through a TCP socket instead of reading the packets from a PCAP file.

A simple way to create a PCAP-over-IP server is to simply read a PCAP file into a netcat listener, like this:

nc -l 57012 < sniffed.pcap

The packets in “sniffed.pcap” can then be read remotely using PCAP-over-IP, for example with tshark like this (replace with the IP of the netcat listener):

nc 57012 | tshark -r -

But there’s an even simpler way to read PCAP-over-IP with Wireshark and tshark, which doesn’t require netcat.

wireshark -k -i TCP@
tshark -i TCP@

The Wireshark name for this input method is “TCP socket” pipe interface, which is available in Linux, Windows and macOS builds of Wireshark as well as tshark.

Live Remote Sniffing

Sniffed traffic can be read remotely over PCAP-over-IP in real-time simply by forwarding a PCAP stream with captured packets to netcat like this:

tcpdump -U -w - not tcp port 57012 | nc -l 57012
dumpcap -P -f "not tcp port 57012" -w - | nc -l 57012
PCAP-over-IP with tcpdump, netcat and tshark

Tcpdump is not available for Windows, but dumpcap is since it is included with Wireshark.

Note how TCP port 57012 is purposely filtered out using BPF when capturing in order to avoid a snowball effect, where the PCAP-over-IP traffic otherwise gets sniffed and re-transmitted through the PCAP-over-IP stream, which again gets sniffed etc.

Reading PCAP-over-IP with NetworkMiner

We added PCAP-over-IP support to NetworkMiner in 2011 as part of NetworkMiner 1.1, which was actually one year before the TCP socket sniffing feature was included in Wireshark.

Live remote sniffing with NetworkMiner 2.7.3 using PCAP-over-IP

Image: Live remote sniffing with NetworkMiner 2.7.3 using PCAP-over-IP

NetworkMiner can also be configured to listen for incoming PCAP-over-IP connections, in which case the sniffer must connect to the machine running NetworkMiner like this:
tcpdump -U -w - not tcp port 57012 | nc 57012

This PCAP-over-IP feature is actually the recommended method for doing real-time analysis of live network traffic when running NetworkMiner in Linux or macOS, because NetworkMiner’s regular sniffing methods are not available on those platforms.

Reading Decrypted TLS Traffic from PolarProxy


One of the most powerful use-cases for PCAP-over-IP is to read decrypted TLS traffic from PolarProxy. When PolarProxy is launched with the argument “--pcapoverip 57012” it starts a listener on TCP port 57012, which listens for incoming connections and pushes a real-time PCAP stream of decrypted TLS traffic to each client that connects. PolarProxy can also make active outgoing PCAP-over-IP connections to a specific IP address and port if the “--pcapoveripconnect <host>:<port>” argument is provided.

In the video PolarProxy in Windows Sandbox I demonstrate how decrypted TLS traffic can be viewed in NetworkMiner in real-time with help of PCAP-over-IP. PolarProxy’s PCAP-over-IP feature can also be used to read decrypted TLS traffic from PolarProxy with Wireshark as well as to send decrypted TLS traffic from PolarProxy to Arkime (aka Moloch).

Replaying PCAP-over-IP to an Interface

There are lots of great network monitoring products and intrusion detection systems that don’t come with a built-in PCAP-over-IP implementation, such as Suricata, Zeek, Security Onion and Packetbeat, just to mention a few. These products would greatly benefit from having access to the decrypted TLS traffic that PolarProxy can provide. Luckily we can use netcat and tcpreplay to replay packets from a PCAP-over-IP stream to a network interface like this:

nc localhost 57012 | tcpreplay -i eth0 -t -

But for permanent installations we recommend creating a dedicated dummy interface, to which the traffic gets replayed and sniffed, and then deploy a systemd service that performs the replay operation. See our blog post Sniffing Decrypted TLS Traffic with Security Onion for an example on how to deploy such a systemd service. In that blog post we show how decrypted TLS traffic from PolarProxy can be replayed to a local interface on a Security Onion machine, which is being monitored by Suricata and Zeek.

Nils Hanke has also compiled a detailed documentation on how decrypted TLS packets from PolarProxy can be replayed to Packetbeat and Suricata with help of tcpreplay.

In these setups netcat and tcpreplay act as a generic glue between a PCAP-over-IP service and tools that can sniff packets on a network interface, but there are a few drawbacks with this approach. One drawback is that tcpreplay requires root privileges in order to replay packets to an interface. Another drawback is that extra complexity is added to the solution and two additional single point of failures are introduced (i.e. netcat and tcpreplay). Finally, replaying packets to a network interface increases the risk of packet drops. We therefore hope to see built-in PCAP-over-IP implementations in more network monitoring solutions in the future!

FAQ for PCAP-over-IP

Q: Why is it called “PCAP-over-IP” and not “PCAP-over-TCP”?

Good question, we actually don’t know since we didn’t come up with the name. But in theory it would probably be feasible to read a PCAP stream over UDP or SCTP as well.

Q: What is the standard port for PCAP-over-IP?

There is no official port registered with IANA for PCAP-over-IP, but we’ve been using TCP 57012 as the default port for PCAP-over-IP since 2011. The Wireshark implementation, on the other hand, uses TCP port 19000 as the default value.

Q: Which software comes with built-in PCAP-over-IP servers or clients?

The ones we know of are: Arkime, NetworkMiner, PolarProxy, tshark and Wireshark.

Q: Is there some way to encrypt the PCAP-over-IP transmissions?

Yes, we recommend encrypting PCAP-over-IP sessions with TLS when they are transmitted across a non-trusted network. NetworkMiner’s PCAP-over-IP implementation comes with a “Use SSL” checkbox, which can be used to receive “PCAP-over-TLS”. You can replace netcat with socat or ncat in order to establish a TLS encrypted connection to NetworkMiner.

Q: Is there a tool that can aggregate multiple PCAP-over-IP streams into one?

No, none that we’re aware of. However, multiple PCAP-over-IP streams can be merged into one by specifying multiple PCAP-over-IP interfaces in dumpcap and then forwarding that output to a netcat listener, like this:

dumpcap -i TCP@ -i TCP@ -w - | editcap -F pcap - - | nc -l 57012

Posted by Erik Hjelmvik on Monday, 15 August 2022 08:05:00 (UTC/GMT)

Tags: #PCAP-over-IP#PCAP#tcpdump#Wireshark#tshark#NetworkMiner#PolarProxy#Suricata#Zeek#Arkime#tcpreplay#netcat#ASCII-art

Share: Facebook   Twitter   Reddit   Hacker News Short URL:

Sniffing Decrypted TLS Traffic with Security Onion

Wouldn't it be awesome to have a NIDS like Snort, Suricata or Zeek inspect HTTP requests leaving your network inside TLS encrypted HTTPS traffic? Yeah, we think so too! We have therefore created this guide on how to configure Security Onion to sniff decrypted TLS traffic with help of PolarProxy.

Network drawing with Clients, SecurityOnion and the Internet

PolarProxy is a forward TLS proxy that decrypts incoming TLS traffic from clients, re-encrypts it and forwards it to the server. One of the key features in PolarProxy is the ability to export the proxied traffic in decrypted form using the PCAP format (a.k.a. libpcap/tcpdump format). This makes it possible to read the decrypted traffic with external tools, without having to perform the decryption again. It also enables packet analysis using tools that don't have built-in TLS decryption support.

This guide outlines how to configure PolarProxy to intercept HTTPS traffic and send the decrypted HTTP traffic to an internal network interface, where it can be sniffed by an IDS.

STEP 1 ☆ Install Ubuntu

Download and install the latest SecurityOnion ISO image, but don't run the "Setup" just yet.

STEP 2 ☆ Add a Dummy Network Interface

Add a dummy network interface called "decrypted", to which decrypted packets will be sent.

ip link add decrypted type dummy
ip link set decrypted arp off up
Add the commands above to /etc/rc.local before "exit 0" to have the network interface automatically configured after reboots.

dummy interface in rc.local

STEP 3 ☆ Install Updates

Install updates in Security Onion by running "sudo soup".

STEP 4 ☆ Run the Security Onion Setup

Run the Security Onion setup utility by double-clicking the "Setup" desktop shortcut or executing "sudo sosetup" from a terminal. Follow the setup steps in the Production Deployment documentation and select "decrypted" as your sniffing interface.

Sniffing Interface Selection Window

Reboot and run Setup again to continue with the second phase of Security Onion's setup. Again, select "decrypted" as the interface to be monitored.

STEP 5 ☆ Install PolarProxy Service

Download and install PolarProxy:

sudo adduser --system --shell /bin/bash proxyuser
sudo mkdir /var/log/PolarProxy
sudo chown proxyuser:root /var/log/PolarProxy/
sudo chmod 0775 /var/log/PolarProxy/

sudo su - proxyuser
mkdir ~/PolarProxy
cd ~/PolarProxy/
curl | tar -xzf -

sudo cp /home/proxyuser/PolarProxy/PolarProxy.service /etc/systemd/system/PolarProxy.service

Edit /etc/systemd/system/PolarProxy.service and add "--pcapoverip 57012" at the end of the ExecStart command.

--pcapoverip 57012 in PolarProxy.service

Start the PolarProxy systemd service:

sudo systemctl enable PolarProxy.service
sudo systemctl start PolarProxy.service

STEP 6 ☆ Install Tcpreplay Service

The decrypted traffic can now be accessed via PolarProxy's PCAP-over-IP service on TCP 57012. We can leverage tcpreplay and netcat to replay these packets to our dummy network interface in order to have them picked up by Security Onion.

nc localhost 57012 | tcpreplay -i decrypted -t -
However, it's better to create a systemd service that does this automatically on bootup. We therefore create a file called /etc/systemd/system/tcpreplay.service with the following contents:
Description=Tcpreplay of decrypted traffic from PolarProxy

ExecStart=/bin/sh -c 'nc localhost 57012 | tcpreplay -i decrypted -t -'


Start the tcpreplay systemd service:

sudo systemctl enable tcpreplay.service
sudo systemctl start tcpreplay.service

STEP 7 ☆ Add firewall rules

Security Onion only accepts incoming connections on TCP 22 by default, we also need to allow connections to TCP port 10443 (proxy port), and 10080 (root CA certificate download web server). Add allow rules for these services to the Security Onion machine's firewall:

sudo ufw allow in 10443/tcp
sudo ufw allow in 10080/tcp

Verify that the proxy is working by running this curl command on a PC connected to the same network as the Security Onion machine:

curl --insecure --connect-to[SecurityOnionIP]:10443
Note: You can even perform this test from a Win10 PC, since curl is included with Windows 10 version 1803 and later.

Add the following lines at the top of /etc/ufw/before.rules (before the *filter section) to redirect incoming packets on TCP 443 to PolarProxy on port 10443.

-A PREROUTING -i enp0s3 -p tcp --dport 443 -j REDIRECT --to 10443

Note: Replace "enp0s3" with the Security Onion interface to which clients will connect.

After saving before.rules, reload ufw to activate the port redirection:

sudo ufw reload

Verify that you can reach the proxy on TCP 443 with this command:

curl --insecure --resolve[SecurityOnionIP]

STEP 8 ☆ Redirect HTTPS traffic to PolarProxy

It's now time to configure a client to run its HTTPS traffic through PolarProxy. Download and install the PolarProxy X.509 root CA certificate from PolarProxy's web service on TCP port 10080:


Install the certificate in the operating system and browser, as instructed in the PolarProxy documentation.

You also need to forward packets from the client machine to the Security Onion machine running PolarProxy. This can be done either by configuring a local NAT rule on each monitored client (STEP 8.a) or by configuring the default gateway's firewall to forward HTTPS traffic from all clients to the proxy (STEP 8.b).

STEP 8.a ☆ Local NAT

Use this firewall rule on a Linux client to configure it to forward outgoing HTTPS traffic to the Security Onion machine:

sudo iptables -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to [SecurityOnionIP]

STEP 8.b ☆ Global NAT Network drawing Firewall, PolarProxy, Clients

If the client isn't running Linux, or if you wanna forward HTTPS traffic from a whole network to the proxy, then apply the following iptables rules to the firewall in front of the client network. See "Routing Option #2" in the PolarProxy documentation for more details.

  1. Add a forward rule on the gateway to allow forwarding traffic to our PolarProxy server:
    sudo iptables -A FORWARD -i eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -m state --state NEW -j ACCEPT
  2. Add a DNAT rule to forward 443 traffic to PolarProxy on port 10443:
    sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j DNAT --to [SecurityOnionIP]:10443
  3. If the reverse traffic from PolarProxy to the client doesn't pass the firewall (i.e. they are on the same LAN), then we must add this hide-nat rule to fool PolarProxy that we are coming from the firewall:
    sudo iptables -t nat -A POSTROUTING -o eth1 -d [SecurityOnionIP] -p tcp --dport 10443 -j MASQUERADE
For other network configurations, please see the various routing setups in the PolarProxy documentation.

STEP 9 ☆ Inspect traffic in SecurityOnion

Wait for the Elastic stack to initialize, so that the intercepted network traffic becomes available through the Kibana GUI. You can check the status of the elastic initialization with "sudo so-elastic-status".

You should now be able to inspect decrypted traffic in Security Onion using Kibana, Squert, Sguil etc., just as if it was unencrypted HTTP.

Bro HTTP traffic in Kibana Image: Kibana showing HTTP traffic info from decrypted HTTPS sessions

MIME types in Kibana Image: MIME types in Kibana

NIDS alerts in Kibana Image: NIDS alerts from payload in decrypted traffic shown in Kibana

Snort alerts in Squert Image: Snort alerts from decrypted traffic shown in Squert

Security Considerations and Hardening

Security Onion nodes are normally configured to only allow access by SOC/CERT/CSIRT analysts, but the setup described in this blog post requires that "normal" users on the client network can access the PolarProxy service running on the Security Onion node. We therefore recommend installing PolarProxy on a dedicated Security Onion Forward Node, which is configured to only monitor traffic from the proxy.

We also recommend segmenting the client network from the analyst network, for example by using separate network interfaces on the Security Onion machine or putting it in a DMZ. Only the PolarProxy service (TCP 10080 and 10443) should be accessable from the client network.

PolarProxy could be used to pivot from the client network into the analyst network or to access the Apache webserver running on the Security Onion node. For example, the following curl command can be used to access the local Apache server running on the Security Onion machine via PolarProxy:

curl --insecure --connect-to localhost:443:[SecurityOnionIP]:10443 https://localhost/
We therefore recommend adding firewall rules that prevent PolarProxy from accessing the analyst network as well as the local Apache server.

Hardening Steps:

  • Configure the Security Onion node as a Forward Node
  • Segment client network from analyst network
  • Add firewall rules to prevent PolarProxy from accessing services on the local machine and analyst network

For additional info on hardening, please see the recommendations provided by Wes Lambert on the Security-Onion mailing list.

Posted by Erik Hjelmvik on Monday, 20 January 2020 09:40:00 (UTC/GMT)

Tags: #SecurityOnion#Security Onion#PCAP#Bro#Zeek#PolarProxy#Snort#Suricata#TLS#SSL#HTTPS#tcpreplay#PCAP-over-IP#IDS#NIDS#netcat#curl#UFW#ASCII-art

Share: Facebook   Twitter   Reddit   Hacker News Short URL:

Detecting the Pony Trojan with RegEx using CapLoader

This short video demonstrates how you can search through PCAP files with regular expressions (regex) using CapLoader and how this can be leveraged in order to improve IDS signatures.

The EmergingThreats snort/suricata rule mentioned in the video is SID 2014411 “ET TROJAN Fareit/Pony Downloader Checkin 2”.

The header accept-encoding header with quality factor 0 used by the Pony malware is:
Accept-Encoding: identity, *;q=0

And here is the regular expression used to search for that exact header: \r\nAccept-Encoding: identity, \*;q=0\r\n

After recording the video I noticed that the leaked source code for Pony 2.0 actually contains this accept-encoding header as a hard-coded string. Have a look in the redirect.php file, where they set curl’s CURLOPT_HTTPHEADER to this specific string.

Pony using curl to set: Accept-Encoding: identity, *;q=0

Wanna learn more about the intended use of quality factors in HTTP accept headers? Then have a look at section 14.1 of RFC 2616section 5.3.4 of RFC 7231, which defines how to use qvalues (i.e. quality factors) in the Accept-Encoding header.

Finally, I'd like to thank Brad Duncan for running the website, your PCAP files often come in handy!

Update 2018-07-05

I submitted a snort/suricata signature to the Emerging-Sigs mailinglist after publishing this blog post, which resulted in the Emerging Threats signature 2014411 being updated on that same day to include:

content:"|0d 0a|Accept-Encoding|3a 20|identity,|20 2a 3b|q=0|0d 0a|"; http_header;

Thank you @EmergingThreats for the fast turnaround!

Posted by Erik Hjelmvik on Wednesday, 04 July 2018 07:39:00 (UTC/GMT)

Tags: #video#regex#malware#IDS#curl#videotutorial

Share: Facebook   Twitter   Reddit   Hacker News Short URL:

Zyklon Malware Network Forensics Video Tutorial

We are releasing a series of network forensics video tutorials throughout the next few weeks. First up is this analysis of a PCAP file containing network traffic from the "Zyklon H.T.T.P." malware.

Analyzing a Zyklon Trojan with Suricata and NetworkMiner


Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20070302 BonEcho/

Posted by Erik Hjelmvik on Monday, 05 February 2018 07:30:00 (UTC/GMT)

Tags: #Netresec#PCAP#Trojan#video#tutorial#videotutorial#NetworkMiner#SecurityOnion#Suricata#malware#network#forensics#NSM#malware_traffic

Share: Facebook   Twitter   Reddit   Hacker News Short URL:

Covert Man-on-the-Side Attacks


After Edward Snowden exposed NSA's Man-on-the-Side attack capabilities we've started to see IDS signatures that can detect such attacks being released and re-discovered. However, despite these efforts Man-on-the-Side attacks, such as QUANTUM INSERT, can still be carried out without triggering these IDS signatures.

I recently taught a network forensics class in Stockholm. One of the topics covered in this training was how to detect Man-on-the-Side attacks in full content PCAP files.

Man-on-the-Side Explained A Man-on-the-Side (MOTS) attack has the following two characteristics:
  • The attacker can read the traffic and insert new messages, but not to modify or delete messages sent by other participants.
  • The attacker relies on a timing advantage to make sure that the response he sends to the request of a victim arrives before the legitimate response.

In practice this means that the attacker relies on packet injection to insert a TCP packet with a payload to be executed by the victim, such as an HTTP redirect to a malicious web site (source The Intercept). The TCP sequence number of this injected packet will typically be the same as that in the real HTTP response coming from the legitimate web server. Thus, the end node will see two overlapping TCP segments with different application layer data.

In one of the labs, in the network forensics training, students were tasked with finding a Man-on-the-Side attack in a 2.3 GB PCAP dataset. However, the way this MOTS attack was carried out made it invisible to normal signatures designed to detect TCP stream overlaps with different data, such as the Suricata signature 2210050.

alert tcp any any -> any any (msg:"SURICATA STREAM reassembly overlap with different data"; stream-event:reassembly_overlap_different_data; classtype:protocol-command-decode; sid:2210050; rev:2;)

The reason why Suricata and other methods fail to detect this attack is because the injected packet contained both application layer data (an HTTP redirect) and a TCP FIN flag. Upon receiving this spoofed packet the client (victim) followed the redirect as well as closed down its current TCP socket to the web server, by responding with a FIN+ACK packet. Subsequent packets sent by the real web server were then ignored by the client since the TCP socket was already closed when they arrived.

Stream reassembly engines in intrusion detection systems also ignore packets sent after the TCP tear-down, since the TCP session is assumed to be closed at this point. Overlapping TCP segments with different data are therefore not detected by intrusion detection systems when an injected TCP packet carries the FIN flag. I've created an example PCAP file, which illustrate this behavior, called mots-with-fin.pcap (this is not the MOTS attack analyzed in my training). Here's what the PCAP file looks like when analyzed with Tshark:

tshark -r mots-with-fin.pcap -T fields -e ip.src -e ip.dst -e ip.ttl -e tcp.seq -e tcp.flags -e http.response.code -e http.response.phrase 64 189665416  0x0002       54 4114717473 0x0012 64 189665417  0x0010 64 189665417  0x0018       64 4114717474 0x0019 302 Found <--INJECTED 64 189665756  0x0010       54 4114717474 0x0010 64 189665756  0x0011       54 4114717474 0x0018 301 Moved Permanently

Frame number 5 is the injected “302 Found” packet spoofed by the attacker. The TCP flag value 0x19 translates to FIN+PUSH+ACK, which is the attackers attempt to tear-down the TCP connection. The client responds with a FIN+ACK (0x11) in frame 8. The final frame is the real HTTP response coming from the legitimate web server.

Detecting MOTS Attacks

Martin Bruse was one of the guys taking the network forensics class last week. After realizing that there currently doesn't seem to exist any effective method for automatically detecting TCP segment overlaps with different data, regardless of the TCP state, Martin developed a tool called qisniff. This is what it looks like when mots-with-fin.pcap is analyzed with qisniff:

go run qisniff.go -file mots_with_fin.pcap
-> 4114717474
HTTP/1.1 302 Found
Location: //
Content-Length: 0

HTTP/1.1 301 Moved Permanently
Date: Tue, 21 Apr 2015 00:40:01 GMT

In the output above we can see the injected content <A> and the legitimate content from the real web server <B>. What qisniff does is basically reassembling streams and comparing the application layer data in new TCP segments with that in previously received segments. This is a very generic way of detecting any form of packet injection in a TCP stream, regardless if it is done as part of a Quantum Insert attack, an Airpwn injection or some brand new packet injection attack.

Martin's qisniff tool is open sourced under a GPLv2 license and is available on GitHub here:

To run qisniff you need to have Go 1.5 installed as well as gopacket.


We would like to thank Fox-IT for publishing their great blog post Deep dive into QUANTUM INSERT, in which they shed some light on many technical details of Man-on-the-Sida attacks as well as published IDS signatures designed to detect such attacks.

UPDATE 2016-02-02

David Stainton has updated his HoneyBadger tool, which is specifically designed detect TCP injection attacks, so that it now also detects injected TCP packets with the FIN flag set. The update was released on January 31, in update 1457755.

HoneyBadger detecting an injected TCP packet with FIN flag Image: HoneyBadger detecting injected packet in the mots-with-fin.pcap file we released.

UPDATE 2016-10-25

I have now released my own tool called "findject", which is a simple python script that can detect packet injection attacks like QUANTUM INSERT. You can read more about how to detect this type of attacks with findject in my blog post "Detect TCP content injection attacks with findject".

findject logo

Posted by Erik Hjelmvik on Monday, 21 September 2015 08:23:00 (UTC/GMT)

Tags: #MOTS#PCAP#TCP#Suricata#stream

Share: Facebook   Twitter   Reddit   Hacker News Short URL:


NETRESEC on Mastodon:


NETRESEC on Twitter: @netresec