NETRESEC Network Security Blog - Tag : Monitor


SCADA Network Forensics with IEC-104

turbine

A great way to enable digital forensics of control system networks is to implement network security monitoring. Captured network traffic is a great source for evidence when analyzing an attackers steps as he attempts to hack a SCADA system. The newly added support for the IEC-104 protocol in NetworkMiner also allows investigators and incident responders to see what commands the attacker sent to the control system.

We at Netresec recently announced the release of NetworkMiner 1.4, which comes with a parser for the SCADA protocol IEC 60870-5-104 (aka IEC-104). Bringing this Industrial Control System (ICS) protocol into NetworkMiner is a first step to support forensics of compromised ICS networks. The traffic from ICS networks does, of course, need to be captured (sniffed) in order to support network forensics; we are strong supporters of such network monitoring for ICS networks (read our “Monitor those Control System Networks” blog post for more details).


Why monitor ICS networks?

Computer forensics typically involves performing forensic analysis of hard disks. Disk forensics is very effective when analyzing a hard drive from a PC (like an operator workstation), but far more complicated when it is an embedded device like a PLC or RTU that is to be analyzed.

In regard to what was believed to be a hacked SCADA system at a water facility in Illinois, David Marcus from McAfee said:

“My gut tells me that there is greater targeting and wider compromise than we know about. Why? Again, my instincts tell me that there is a lack of cyber forensics and response procedures at most of these facilities. If you do not have cyber forensic capabilities, it is hard to know if you have a cyber intrusion.”

Even though the hack was later shown to just be just a false alarm, David’s point about lacking capabilities for digital forensics and incident response for this type of critical infrastructure still holds true.

Joe Weiss also commented on the same story saying:

“We don't know how many other SCADA systems have been compromised because they don't really have cyber forensics.”

As Joe and David say, the ability to perform digital forensics in SCADA systems is truly lacking today. Our propose with this blog post is to inform control system operators that forensic data/evidence can be easily collected from ICS / SCADA systems by implementing a simple solution for network monitoring with full packet capture.


How to monitor ICS networks

The SCADA network diagram below has been sectioned into multiple security zones according to the zoning principle published by Jens Z, Iiro and me at CIRED 2009 (our zones align nicely with ISA-99 security Levels by the way).

SCADA Network with security zones

The purple octagons represent interconnections between zones. Each such interconnection should be secured with perimeter protection, typically by a firewall, but we additionally argue that all network traffic passing through should be captured and stored as pcap files. Storing all network traffic this way makes it possible to perform network forensics on the network traffic after an intrusion is believed to have taken place.

We recommend a very simple setup, where a network tap is used to provide a copy of all traffic to a sniffer. An acceptable alternative to buying a network tap is to configure a monitor / SPAN port on a switch (see our sniffing tutorial “Intercepting Network Traffic” for more details on how to choose sniffing hardware).

Connection of network tap and sniffer

Our recommended solution for the sniffer is to install FreeBSD with dumpcap (part of the net/tshark ports package). An even easier solution is to install Doug BurksSecurity Onion, which is a Linux distro built especially for network security monitoring. More about configuring a sniffer can be found in our second sniffing tutorial titled “Dumping Network Traffic to Disk”.


Analyzing captured IEC 104 traffic

Let’s assume the file 090813_diverse.pcap from pcapr contains network traffic from a suspected security breach at a hydro-power plant. Let’s also assume that parameter 4821 (i.e. IOA 4821 in IEC-104 language) controls the floodgates of the plant’s dam, where setting a value greater than 0% for this parameter would mean opening the floodgates.

By loading the pcap file into NetworkMiner and selecting the “parameters” tab we can see a nice log of all IEC-104 communication.

NetworkMiner 1.4.1 with Parameters tab

NOTE: We’ve hidden several fields (like IP, port, time etc) in the screenshot above in order to make it fit.

The following timeline can be extracted from the list of events provided by NetworkMiner:

  • Frame 154 - The attacker sends command to set IOA 4821 to 50.354%
  • Frame 156 - The RTU confirms the request
  • Frame 162 - The RTU reports that the requested command has been successfully completed, i.e. floodgates are now open!

Open dam gates by David Baird

More ICS protocols

Would you like to see more ICS protocols in NetworkMiner? We’d be happy to implement protocols like DNP3, MODBUS, ICCP, Siemens S7, IEC 61850, etc. if you can provide us with captured network traffic! Please send an email to info[at]netresec.com if you are interested!

Posted by Erik Hjelmvik on Thursday, 30 August 2012 12:03:00 (UTC/GMT)

Tags: #Forensics #ICS #SCADA #control system #Network #Sniff #Capture #Monitor #pcap

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=1284162


Sniffing Tutorial part 2 - Dumping Network Traffic to Disk

disk II

This blog post is the second part of a two-part tutorial that shows how to sniff network traffic. The first part covered how to intercept the traffic, i.e. how to get the packets to arrive to your network card. This second part covers how to best capture the network traffic to disk once you've managed to have them sent to your network card.

The first thing I recommend you to do before you start sniffing is to ensure that your sniffer computer stays quiet, so that you don't pollute the sniffed traffic with packets from your sniffer computer. How to do this varies between Windows, Linux and FreeBSD, so I will briefly cover how to silence all three OS's.

Silencing Windows

Windows is normally quite chatty on the network, but there is a simple trick that makes it shut up without disconnecting it from the network. Simply bring up the properties window for your network adapter and un-check all clients, services and protocols from the list of used items.

Local Area Connection Properties

Silencing Linux

I don't know any good way to disable the TCP/IP stack in Linux similar to what we just did to Windows. But by disabling ARP and not giving the interface any IP address the risk of leaking packets can be reduced significantly. To achieve this you'll also need to set the interface to promiscuous mode and disable IPv6. Here's an example of how this can be done in Debian/Ubuntu:

# ifconfig eth0 -arp up
# ifconfig eth0 promisc
# ip addr del 192.168.3.4/24 dev eth0
# ifconfig eth0 del fe80::1234:ddff:fe3f:1337/64
However, in general it is better to configure the sniffer interface by modifying /etc/network/interfaces to look as follows:
auto eth0
  iface eth0 inet manual
  up ifconfig $IFACE -arp up
  up ip link set $IFACE promisc on
  down ip link set $IFACE promisc off
  down ifconfig $IFACE down

For more details on shuting up your Linux machine I suggest you take a look at the “Undetectable Sniffing On Ethernet” blog post on the Ask Apache blog. I also recommend reading the Ubuntu instructions at the NetworkConfiguration wiki page for Security Onion.

Silencing FreeBSD

One of the many great things about FreeBSD is that it is really easy to configure a network interface to be silent. All you need to do is to issue this single command:

# ifconfig em0 monitor up

Read Richard Bejtlich's blog post “Ifconfig Monitor Option” for more info about this command.

Sniff with dumpcap

The most common software used for sniffing traffic is undoubtedly Wireshark, other common tools are tshark and tcpdump. All these three tools parse the packets as they come in and show data about the sniffed packets in the GUI or on the command line. This isn't desirable when doing network forensics since these application could use up too much resources on parsing the incoming packets, which during traffic peaks can lead to packets being dropped. There might also be bugs in the packet parsing code that can cause the application to crash completely, there have been plenty of such bugs in Wireshark for example.

A much more reliable way to sniff traffic is therefore to use an application that is designed to reliably dump traffic to disk, without trying to parse or analyze the contents of the sniffed frames. One such application is dumpcap, which is a command line tool developed by the Wireshark crew. In fact, as of version 1.4.0 of Wireshark both tshark and Wireshark run dumpcap in the background rather than doing the sniffing themselves. Another great thing about dumpcap is that it is available on Windows as well as on Linux and FreeBSD. My recommendation is to run it under FreeBSD since I consider it to be more reliable than the other operating systems. In order to get dumcap you need to download either Wirehsark or tshark since dumpcap doesn't have an installation package of its own.

Dumpcap supports capture filtering with BPF, but I recommend that you sniff ALL packets if you are authorized to do so and have enough disk space. It is much better to do the filtering later on in the analysis phase. The feature I like best in dumpcap is the ring buffer functionality, where it will continue writing incoming frames to a new pcap file once the old one has reached a specific size limit. In order to have dumpcap write sniffed frames to disk and roll over to a new file when it reaches 100.000 kB you'd run this command:

# dumpcap -i em0 -w wiretap.pcap -b filesize:100000

When you've dumped the traffic to disk is when the fun starts, i.e. analyzing the pcap files. You can read more analyzing pcap files in my blog posts about the TCP/IP Weapons School Sample Lab, DFRWS 2009 Challenge, Webmail Information Leakage and Command-line forensics.

Posted by Erik Hjelmvik on Friday, 11 March 2011 16:56:00 (UTC/GMT)

Tags: #dumpcap #Wireshark #tshark #dump #pcap

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=1135E10


Sniffing Tutorial part 1 - Intercepting Network Traffic

A cow's nose

This blog post is the first part of a two-part tutorial that shows how to sniff network traffic. This first part covers how to intercept the traffic, i.e. how to get the packets to arrive to your network card. The second part covers how to best capture the network traffic to disk once you've managed to have them sent to your network card.

I divide the field of network forensics and network security monitoring into two disciplines; capturing (a.k.a. sniffing) the traffic and analyzing the captured traffic. Analyzing the network traffic can be done on many different levels, from the novice's level where IP addresses and layer 7 protocols are listed to the expert who understands all protocol layers and can investigate traffic packet-by-packet. Network traffic analysis is a field where there is always something more to learn and where nobody, not even an expert like Richard Bejtlich, knows everything. Capturing traffic, on the other hand, is more like a craftsmanship; it takes a while to learn how to do it properly, but there is not much new to learn once you've gotten the hang of it. Being able to capture network traffic reliably is, however, essential in order to perform network traffic analysis or network forensics, which is why I decided to write this sniffing tutorial.

Many “n00bs” fire up Wireshark on their own PC expecting to be able to sniff all traffic passing through an Ethernet network. But the 90's are long gone, and all hubs have been replaced by switches, so your NIC nowadays only gets broadcast packets and packets addressed to your NIC. So if you wanna capture traffic from other hosts on the network you somehow need to force that traffic to passes by your NIC. I will here explain some of the most usual ways to achieve this.

Use a legacy “dumb” hub

4 port Netgear Ethernet hub

The good ol' dumb hubs (a.k.a. active hubs) are hard to find these days, but if you're fortunate enough to own one of those old jewels then make sure to hold on to it. You might also get lucky and find a used hub on ebay. These beauties repeat all Ethernet frames arriving at one port to all the other ports on the hub, which is just what we want in order to sniff all the traffic. Just place the hub at a choke-point in your network, for example between your firewall and the rest of the network, and connect your sniffer PC directly to the hub (see my ASCII image below).

Hub in choke-point

One downside of using a hub is that all these extra packets being sent on all the hub's ports increase the risk of packet collisions, i.e. when two nodes on a shared media network are transmitting at the same time. Every time a collision occurs on an Ethernet network the nodes on the shared media segment will pause for a random time before they continue sending data, this functionality is called CSMA/CD. For TCP based protocols such collisions result in lots of ugly retransmissions, while the data in collided UDP packets never will reach their destination since UDP doesn't support retransmissions.

Configure a monitor port on a managed switch

Network switches

A much better solution than using a hub is to configure a monitor port on a managed switch, from which you wish to tap some traffic. Monitor ports have different names depending on the vendor, Cisco call theirs SPAN ports, others use the term “port mirroring”. A monitor port is typically configured so that it copies all sent and received frames for a particular port to a monitor port, which is where you connect your sniffer. You can normally also configure the switch to copy traffic from all ports to the monitor port, but this can cause duplicate packets since each packet will be copied both when coming in on one port as well as when it goes out on another port. It is therefore usually better to monitor only the uplink port if you wish to monitor all hosts connected to a switch.

The major downside of sniffing traffic of a monitor port is that the primary functionality of a switch is to forward traffic from the sender to the receiver, not to send copies of that traffic to a monitor port. This means that if there is a high load on the switch it will prioritize sending the received frames to the recipient over copying them to the monitor port. You therefore run the risk of not always getting all transmitted packets to your monitor port, which might be acceptable in some situations but isn't acceptable in a network forensics investigation. Finding out if your monitor port has missed some packets isn't easy, but Charles Smutz explains how this can be done in his blog post “Flushing out Leaky Taps”.

Network Tap

The most reliable way to sniff traffic is to use a network tap. A network tap is a “bump-in-the-wire” device designed only to copy traffic passing through it to a monitor port. You typically insert a network tap inline between two nodes in a network, such as between your firewall and your first switch. A good network tap will cost you at least 1.000 EUR, but those are money well spent if you wanna be able to perform reliable network monitoring.

If you're planning to buy a network tap I recommend getting a link aggregation tap, which can merge uplink and downlink traffic to one single monitor port. This way you don't need two network interfaces on your sniffer PC. Both major tap vendors Net Optics and VSS Monitoring sell aggregation taps.

A downside of using a network tap compared to configuring a monitor port is that there will be a short downtime on the network while connecting and disconnecting the tap. I therefore recommend to install network taps when you have a service window. And when you have a tap connected at a good choke-point in your network I recommend that you leave it there after you're done, since you never know when you might need to come back and sniff some more.

ARP poisoning

A technique commonly used by hackers and penetration testers for getting hold of traffic in a switched environment is to use ARP poisoning. Basically ARP poisoning is a technique where two hosts on a network are tricked into sending packets destined for each other to a sniffer machine on the network. I'm not gonna describe this method in any more detail than so since ARP poisoning is a technique that I don't recommend using. Tools used for performing ARP poisoning are hacker tools like Cain & Abel and Ettercap. Another similar technique that I also don't recommend using is overflowing the CAM table.

Still not bored?

If you wanna learn even more about setting up monitor ports and sniffing traffic I recommend that you take a look at Betty DuBois' presentation titled “BU-11: SPAN/Mirror/Monitor vs. Taps: When should I use what, and why should I care?” from Sharkfest '09. The good people of LoveMyTool have provided a video recording of Betty's talk at http://lovemytool.blip.tv/file/2277497/.

There is also a great list of various network tapping options over at the Argus website.

Continue reading part 2 of this sniffing tutorial, where I cover how to dump the traffic to disk.

Posted by Erik Hjelmvik on Friday, 11 March 2011 16:52:00 (UTC/GMT)

Tags: #Sniff #Capture #Monitor

More... Share  |  Facebook   Twitter   Reddit   Hacker News Short URL: http://netres.ec/?b=113EA66

twitter

NETRESEC on Twitter

Follow @netresec on twitter:
» twitter.com/netresec


book

Recommended Books

» The Practice of Network Security Monitoring, Richard Bejtlich (2013)

» Applied Network Security Monitoring, Chris Sanders and Jason Smith (2013)

» Network Forensics, Sherri Davidoff and Jonathan Ham (2012)

» The Tao of Network Security Monitoring, Richard Bejtlich (2004)

» Practical Packet Analysis, Chris Sanders (2017)

» Windows Forensic Analysis, Harlan Carvey (2009)

» TCP/IP Illustrated, Volume 1, Kevin Fall and Richard Stevens (2011)

» Industrial Network Security, Eric D. Knapp and Joel Langill (2014)