NETRESEC Network Security Blog - Tag : Protocol Identification


Port Independent Protocol Detection

Protocol Alphabet Soup by ThousandEyes

Our heavy-duty PCAP analyzer CapLoader comes with a feature called ”Port Independent Protocol Identification”, a.k.a. PIPI (see Richard Bejtlich's PIPI blog post from 2006). Academic research in the Traffic Measurement field often use the term ”Traffic Classification”, which is similar but not the same thing. Traffic Classification normally group network traffic in broad classes, such as Email, Web, Chat or VoIP. CapLoader, on the other hand, identifies the actual application layer protocol used in each flow. So instead of classifying a flow as ”VoIP” CapLoader will tell you if the flow carries SIP, Skype, RTP or MGCP traffic. This approach is also known as “Dynamic Protocol Detection”.

Being able to identify application layer protocols without relying on the TCP or UDP port number is crucial when analyzing malicious traffic, such as malware Command-and-Control (C2) communication, covert backdoors and rouge servers, since such communication often use services on non-standard ports. Some common examples are:

  • Many botnet C2 protocols communicate over port TCP 443, but using a proprietary protocol rather than HTTP over SSL.
  • Backdoors on hacked computers and network devices typically wither run a standard service like SSH on a port other than 22 in order to hide.
  • More advanced backdoors use port knocking to run a proprietary C2 protocol on a standard port (SYNful knock runs on TCP 80).

This means that by analyzing network traffic for port-protocol anomalies, like an outgoing TCP connection to TCP 443 that isn't SSL, you can effectively detect intrusions without having IDS signatures for all C2 protocols. This analysis technique is often used when performing Rinse-Repeat Intrusion Detection, which is a blacklist-free approach for identifying intrusions and other form of malicious network traffic. With CapLoader one can simply apply a BPF filter like “port 443” and scroll through the displayed flows to make sure they are all say “SSL” in the Protocol column.

CapLoader detects non-SSL traffic to 1.web-counter.info Image: Miuref/Boaxxe Trojan C2 traffic to "1.web-counter[.]info" on TCP 443 doesn't use SSL (or HTTPS)

Statistical Analysis

CapLoader relies on statistical analysis of each TCP, UDP and SCTP session's behavior in order to compare it to previously computed statistical models for known protocols. These statistical models are generated using a multitude of metrics, such as inter-packet delays, packet sizes and payload data. The port number is, on the other hand, a parameter that is intentionally not used by CapLoader to determine the application layer protocol.

The PIPI/Dynamic Protocol Detection feature in CapLoader has been designed to detect even encrypted and obfuscated binary protocols, such as Tor and Encrypted BitTorrent (MSE). These protocols are designed in order to deceive protocol detection mechanisms, and traditional signature based protocol detection algorithms can't reliably detect them. The statistical approach employed by CapLoader can, on the other hand, actually detect even these highly obfuscated protocols. It is, however, important to note that being a statistical method it will never be 100% accurate. Analysts should therefore not take for granted that a flow is using the protocol stated by CapLoader. There are some situations when it is very difficult to accurately classify an encrypted protocol, such as when the first part of a TCP session is missing in the analyzed data. This can occur when there is an ongoing session that was established before the packet capture was started.


Identified Protocols

The following protocols are currently available for detection in CapLoader's protocol database:

AOL Instant Messenger
BACnet
BitTorrent
BitTorrent Encrypted - MSE
CCCam
CUPS
DAYTIME
DHCP
DHCPv6
Diameter
DirectConnect
DNS
Dockster
DropBox LSP
eDonkey
eDonkey Obfuscated
EtherNet-IP
FTP
Gh0st RAT
Gnutella
Groove LAN DPP
HSRP
HTTP
IMAP
IRC
ISAKMP
iSCSI
JavaRMI
Kelihos
Kerberos
L2TP
LDAP
LLC
Meterpreter
MgCam
MGCP
MikroTik NDP
Modbus TCP
MSN Messenger
MS RPC
MS-SQL
MySQL
NAT-PMP
NetBIOS Datagram Service
NetBIOS Name Service
NetBIOS Session Service
NetFlow
NTP
OsCam
Pcap-over-IP
Poison Ivy RAT
POP3
QUIC
Ramnit
Reverse Shell
RTCP
RTP
RTSP
Shell
SIP
Skype
SLP
SMTP
SNMP
Socks
SopCast P2P
Spotify P2P
Spotify Server
SSH
SSL
Syslog
TeamViewer
TeamViewer UDP
Telnet
Teredo
TFTP
TFTP Data
TPKT
VNC
WS-Discovery
XMPP Jabber
ZeroAccess
Zeus TCP
Zeus UDP

The list of implemented protocols is constantly being increased with new protocols.


PIPI in NetworkMiner

NetworkMiner Logo

NetworkMiner Professional, which is the commercial version of NetworkMiner, also comes with an implementation of our protocol detection mechanism. Even though NetworkMiner Professional doesn't detect as many protocols as CapLoader, the PIPI feature built into NetworkMiner Pro still helps a lot when analyzing HTTP traffic on ports other that 80 or 8080 as well as in order to reassemble files downloaded from FTP or TFTP servers running on non-standard ports.

 

Posted by Erik Hjelmvik on Tuesday, 06 October 2015 09:05:00 (UTC/GMT)

Tags: #Protocol Identification #CapLoader #VoIP #SIP #RTP #TOR #SSL #PIPI #PCAP #NetworkMiner

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=15A1776


Detecting TOR Communication in Network Traffic

The anonymity network Tor is often misused by hackers and criminals in order to remotely control hacked computers. In this blog post we explain why Tor is so well suited for such malicious purposes, but also how incident responders can detect Tor traffic in their networks.

Yellow onions with cross section. Photo taken by Andrew c

The privacy network Tor (originally short for The Onion Router) is often used by activists and whistleblowers, who wish to preserve their anonymity online. Tor is also used by citizens of countries with censored Internet (like in China, Saudi Arabia and Belarus), in order to evade the online censorship and surveillance systems. Authorities in repressive regimes are therefore actively trying to detect and block Tor traffic, which makes research on Tor protocol detection a sensitive subject.

Tor is, however, not only used for good; a great deal of the traffic in the Tor networks is in fact port scans, hacking attempts, exfiltration of stolen data and other forms of online criminality. Additionally, in December last year researchers at Rapid7 revealed a botnet called “SkyNet” that used Tor for its Command-and-Control (C2) communication. Here is what they wrote about the choice of running the C2 over Tor:

"Common botnets generally host their Command & Control (C&C) infrastructure on hacked, bought or rented servers, possibly registering domains to resolve the IP addresses of their servers. This approach exposes the botnet from being taken down or hijacked. The security industry generally will try to take the C&C servers offline and/or takeover the associated domains
[...]
What the Skynet botnet creator realized, is that he could build a much stronger infrastructure at no cost just by utilizing Tor as the internal communication protocol, and by using the Hidden Services functionality that Tor provides."


Tor disguised as HTTPS

Tor doesn't just provide encryption, it is also designed to look like normal HTTPS traffic. This makes Tor channels blend in quite well with normal web surfing traffic, which makes Tor communication difficult to identify even for experienced incident responders. As an example, here is how tshark interprets a Tor session to port TCP 443:

$ tshark -nr tbot_2E1814CCCF0.218EB916.pcap | head
1 0.000000 172.16.253.130 -> 86.59.21.38 TCP 62 1565 > 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1
2 0.126186 86.59.21.38 -> 172.16.253.130 TCP 60 443 > 1565 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460
3 0.126212 172.16.253.130 -> 86.59.21.38 TCP 54 1565 > 443 [ACK] Seq=1 Ack=1 Win=64240 Len=0
4 0.127964 172.16.253.130 -> 86.59.21.38 SSL 256 Client Hello
5 0.128304 86.59.21.38 -> 172.16.253.130 TCP 60 443 > 1565 [ACK] Seq=1 Ack=203 Win=64240 Len=0
6 0.253035 86.59.21.38 -> 172.16.253.130 TLSv1 990 Server Hello, Certificate, Server Key Exchange, Server Hello Done
7 0.259231 172.16.253.130 -> 86.59.21.38 TLSv1 252 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
8 0.259408 86.59.21.38 -> 172.16.253.130 TCP 60 443 > 1565 [ACK] Seq=937 Ack=401 Win=64240 Len=0
9 0.379712 86.59.21.38 -> 172.16.253.130 TLSv1 113 Change Cipher Spec, Encrypted Handshake Message
10 0.380009 172.16.253.130 -> 86.59.21.38 TLSv1 251 Encrypted Handshake Message

A Tor session to TCP port 443, decoded by tshark as if it was HTTPS

The thsark output above looks no different from when a real HTTPS session is being analyzed. So in order to detect Tor traffic one will need to apply some sort of traffic classification or application identification. However, most implementations for protocol identification rely on either port number inspection or protocol specification validation. But Tor often communicate over TCP 443 and it also follows the TLS protocol spec (RFC 2246), because of this most products for intrusion detection and deep packet inspection actually fail at identifying Tor traffic. A successful method for detecting Tor traffic is to instead utilize statistical analysis of the communication protocol in order to tell different SSL implementations apart. One of the very few tools that has support for protocol identification via statistical analysis is CapLoader.

CapLoader provides the ability to differentiate between different types of SSL traffic without relying on port numbers. This means that Tor sessions can easily be identified in a network full of HTTPS traffic.


Analyzing the tbot PCAPs from Contagio

@snowfl0w provides some nice analysis of the SkyNet botnet (a.k.a. Trojan.Tbot) at the Contagio malware dump, where she also provides PCAP files with the network traffic generated by the botnet.

The following six PCAP files are provided via Contagio:

  1. tbot_191B26BAFDF58397088C88A1B3BAC5A6.pcap (7.55 MB)
  2. tbot_23AAB9C1C462F3FDFDDD98181E963230.pcap (3.24 MB)
  3. tbot_2E1814CCCF0C3BB2CC32E0A0671C0891.pcap (4.08 MB)
  4. tbot_5375FB5E867680FFB8E72D29DB9ABBD5.pcap (5.19 MB)
  5. tbot_A0552D1BC1A4897141CFA56F75C04857.pcap (3.97 MB) [only outgoing packets]
  6. tbot_FC7C3E087789824F34A9309DA2388CE5.pcap (7.43 MB)

Unfortunately the file “tbot_A055[...]” only contains outgoing network traffic. This was likely caused by an incorrect sniffer setup, such as a misconfigured switch monitor port (aka SPAN port) or failure to capture the traffic from both monitor ports on a non-aggregating network tap (we recommend using aggregation taps in order to avoid these types of problems, see our sniffing tutorial for more details). The analysis provided here is therefore based on the other five pcap files provided by Contagio.

Here is a timeline with relative timestamps (the frame timestamps in the provided PCAP files were way of anyway, we noticed an offset of over 2 months!):

  • 0 seconds : Victim boots up and requests an IP via DHCP
  • 5 seconds : Victim perform a DNS query for time.windows.com
  • 6 seconds : Victim gets time via NTP
    ---{malware most likely gets executed here somewhere}---
  • 22 seconds : Victim performs DNS query for checkip.dyndns.org
  • 22 seconds : Victim gets its external IP via an HTTP GET request to checkip.dyndns.org
  • 23 seconds : Victim connects to the Tor network, typically on port TCP 9001 or 443
    ---{lots of Tor traffic from here on}---

This is what it looks like when one of the tbot pcap files has been loaded into CapLoader with the “Identify protocols” feature activated:
CapLoader detecting Tor protocol
CapLoader with protocol detection in action - see “TOR” in the “Sub_Protocol” column

Notice how the flows to TCP ports 80, 9101 and 443 are classified as Tor? The statistical method for protocol detection in CapLoader is so effective that CapLoader actually ignores port numbers altogether when identifying the protocol. The speed with which CapLoader parses PCAP files also enables analysis of very large capture files. A simple way to detect Tor traffic in large volumes of network traffic is therefore to load a capture file into CapLoader (with “Identify protocols” activated), sort the flows on the “Sub_Protocol” column, and scroll down to the flows classified as Tor protocol.


Beware of more Tor backdoors

Most companies and organizations allow traffic on TCP 443 to pass through their firewalls without content inspection. The privacy provided by Tor additionally makes it easy for a botnet herder to control infected machines without risking his identity to be revealed. These two factors make Tor a perfect fit for hackers and online criminals who need to control infected machines remotely.

Here is what Claudio Guarnieri says about the future use of Tor for botnets in his Rapid7 blog post:

“The most important factor is certainly the adoption of Tor as the main communication channel and the use of Hidden Services for protecting the backend infrastructure. While it’s surprising that not more botnets adopt the same design, we can likely expect more to follow the lead in the future.”

Incident responders will therefore need to learn how to detect Tor traffic in their networks, not just in order to deal with insiders or rogue users, but also in order to counter malware using it as part of their command-and-control infrastructure. However, as I've shown in this blog post, telling Tor apart from normal SSL traffic is difficult. But making use of statistical protocol detection, such as the Port Independent Protocol Identification (PIPI) feature provided with CapLoader, is in fact an effective method to detect Tor traffic in your networks.

Posted by Erik Hjelmvik on Saturday, 06 April 2013 20:55:00 (UTC/GMT)

Tags: #CapLoader #TOR #Protocol Identification #SSL #TLS #HTTPS #PCAP #PIPI

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=13400BE


CapLoader 1.1 Released

CapLoader Logo Version 1.1 of the super-fast PCAP parsing tool CapLoader is being released today. CapLoader is the ideal tool for digging through large volumes of PCAP files. Datasets in the GB and even TB order can be loaded into CapLoader to produce a clear view of all TCP and UDP flows. CapLoader also provides instantaneous access to the raw packets of each flow, which makes it a perfect preloader tool in order to select and export interesting data to other tools like NetworkMiner or Wireshark.

Drag-and-Drop PCAP from CapLoader to Wireshark
Five flows being extracted from Honeynet.org's SOTM 28 to Wireshark with CapLoader

New functionality in version 1.1

New features in version 1.1 of CapLoader are:

  • PcapNG support
  • Fast transcript of TCP and UDP flows (similar to Wireshark's ”Follow TCP Stream”)
  • Better port agnostic protocol identification; more protocols and better precision (over 100 protocols and sub-protocols can now be identified, including Skype and the C&C protocol of Poison Ivy RAT)
  • A “Hosts” tab containing a list of all transmitting hosts and information about open ports, operating system as well as Geo-IP localization (using GeoLite data created by MaxMind)
  • Gzip compressed capture files can be opened directly with CapLoader
  • Pcap files can be loaded directly from an URL

CapLoader Flow Transcript aka Follow TCP Stream
Flow transcript of Honeynet SOTM 28 pcap file day3.log

Free Trial Version

Another thing that is completely new with version 1.1 of CapLoader is that we now provide a free trial version for download. The CapLoader trial is free to use for anyone and we don't even require trial users to register their email addresses when downloading the software.

There are, of course, a few limitations in the trial version; such as no protocol identification, OS fingerprinting or GeoIP localization. There is also a limit as to how many gigabyte of data that can be loaded with the CapLoader trial at a time. This size limit is 500 GB, which should by far exceed what can be loaded with competing commercial software like Cascade Pilot and NetWitness Investigator.

The professional edition of CapLoader doesn't have any max PCAP limit whatsoever, which allows for terabytes of capture files to be loaded.

CapLoader with TCP and UDP flows view
CapLoader's Flows view showing TCP and UDP flows

CapLoader with Hosts view
CapLoader's Hosts view showing identified hosts on the network

Getting CapLoader

The trial version of CapLoader can be downloaded from the CapLoader product page. The professional edition of CapLoader can be bought at our Purchase CapLoader page.

CapLoader USB flash drive
The CapLoader USB flash drive

Customers who have previously bought CapLoader 1.0 can upgrade to version 1.1 by downloading an update from our customer portal.

For more information about CapLoader please see our previous blog post Fast analysis of large pcap files with CapLoader

Posted by Erik Hjelmvik on Monday, 21 January 2013 11:45:00 (UTC/GMT)

Tags: #CapLoader #PcapNG #PCAP #GB #gigabyte #capture #flow #transcript #free

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: https://netresec.com/?b=1310CE5

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)