NETRESEC Network Security Blog - Tag : SCADA
One of the highlights at this year’s SEC-T conference in Stockholm was Steve Miller’s talk titled "Reversing the TriStation Network Protocol". In this talk Steve covered his quest to better understand the TRITON malware, which had been used in a targeted attack of an industrial control system (ICS). Steve didn’t disclose the type or location of the plant, saying “Don’t ask me who it was, ‘cause I can’t say” when the Q&A started. However, an article in the Wall Street Journal points out that it was a petrochemical plant in Saudi Arabia that had been hacked.
Targeting Safety Instrumented System
The TRITON malware (also called TRISIS) was used to target a safety instrumented system (SIS) from Schneider Electric called Triconex. A SIS is typically not used to control the process of a plant, but rather to detect abnormal operating conditions and safely shut down the industrial process if needed.
I could elaborate a lot regarding the consequences of attacking the SIS, but the good guys from Dragos have already done a great job explaining this in their “TRISIS Malware” report.
Reverse Engineering the ICS Protocol
The communication protocol used by the Triconex controllers is called TriStation, which is a proprietary protocol. This means that there were no publicly available specifications available for the protocol at that time. There was also no Wireshark dissector that could parse TriStation traffic. Nevertheless, Steve’s initial reaction to this was “Awesome, undocumented things are my favorite things!”
Unfortunately Steve wasn’t able to get hold of a single PCAP file with the TriStation network protocol, which made it really difficult to reverse engineer the protocol implementation in the TRITON malware. The only piece of actual TriStation network traffic he was able to get hold of was a hex dump of a TriStation packet in an academic paper.
Armed with only the hexdump and Wireshark’s text2pcap Steve managed to piece together an actual PCAP file containing a single frame with a TriStation packet inside.
As you can see in the image above, Wireshark doesn’t decode any of the application layer data coming from TCP port 1502 (which TriStation uses). He therefore implemented a Wireshark Lua dissector for the TriStation protocol. And some time later the people from Nozomi Networks even implemented a proper Wireshark dissector for the TriStation protocol.
BSI’s ICS-SEC team have now also created Snort IDS rules specifically for the TriStation protocol. These IDS rules trigger on events like:
- Packets sent to the controller from an unauthorized host
- Malicious commands used by the TRITON malware to read and write to the RAM of the SIS controller as well as to execute code
The Importance of Sniffing ICS Traffic
I’ve been trying to convince asset owners, who use ICS in their power plants, factories, water treatment facilities etc, to start capturing the network traffic and storing it as PCAP files for many years now. However, asset owners sometimes try to argue that there is no point in capturing their traffic since it is using a proprietary protocol. Even Ralph Langner has opposed to the idea of capturing ICS network traffic in a blog post, which I have criticized. So, how difficult is it to write a parser for a proprietary protocol?
I have personally implemented support for over 30 application layer protocols in NetworkMiner, but unlike Steve I’ve always had access to at least one PCAP file and some form of documentation of the protocol. However, I’ve found that many real-world protocol implementations don’t follow specifications properly. In these cases I’ve found that having access to PCAP files with real-world network traffic is more important than having a full protocol specification.
Even complex proprietary protocols like the old proprietary Skype protocol has been reverse engineered, so with access to network traffic of a protocol combined with a binary that uses this protocol I’d say that pretty much any network protocol can be reverse engineered.
Steve’s SEC-T talk also proves that ICS protocols are no different, since they too can be reverse engineered without having a protocol specification or RFC.
Capturing network traffic in ICS networks is never wrong. There might not be parsers available today for all the protocols you’re using. But once a parser or IDS signature becomes available for the protocol you’re using, you can simply use that to analyze previously captured network traffic from your ICS network. Also, in the wake of an incident you might actually end up writing a parser (as in the TRITON case) or a custom IDS rule, in which case having historical network traffic from your plant in invaluable!
For more information on this topic I’d suggest reading my blog post titled “Monitor those Control System Networks!” from 2011, which still is highly relevant.
I’m also happy to announce that two PCAP files containing TriStation network traffic have been linked from our list of publicly accessible PCAP files today (see the “SCADA/ICS Network Captures” section).
And remember: PCAP or it didn’t happen!
Posted by Erik Hjelmvik on Friday, 21 September 2018 14:20:00 (UTC/GMT)
4SICS last month and brought back a bunch of PCAP files. Not just any PCAP files, but captured network traffic from the ICS lab that was set up in the Geek Lounge at 4SICS. These PCAP files are now made publicly available here, because captured network traffic from ICS/SCADA networks is a really scarce resource.
4SICS is the the leading Industrial Control System (ICS) security conference in Europe, which brings in speakers and attendees from all around the world. I tought a one-day class on analyzing network traffic as part of the pre-conference training at 4SICS. In this class we analyzed PCAP files containing industrial protocols, such as Modbus/TCP and IEC-104. Unfortunately there aren't many capture files around that carry these protocols, so the ICS analysis part in my class wasn't as advanced as I wanted it to be.
I have been aware of this limited access to ICS traffic for some time now, which is why I decided to work with the 4SICS crew in order to set up a sniffer in the ICS lab at the 4SICS conference. This lab contained devices such as PLCs, RTUs, servers, industrial network equipment (switches, firewalls, etc), which were available for hands-on "testing" by 4SICS attendees.
4SICS ICS Lab. Image Credit: 4SICS
The network TAP vendor Garland were Technology Partners at 4SICS, so I didn't even have to bring a network TAP to the lab. I just connected my sniffer machine and let it record for three days. Chris Sistrunk also joined the sniffing party later in the conference by connecting his SEL-3355, which runs SecurityOnion, to the network TAP.Image Credit: Patrick Nixdorf
The 350MB of network traffic that was captured during the 4SICS conference is now publicly available here:
Posted by Erik Hjelmvik on Wednesday, 04 November 2015 15:45:00 (UTC/GMT)
The Havex backdoor is developed and used by a hacker group called Dragonfly, who are also known as "Energetic Bear" and "Crouching Yeti". Dragonfly is an APT hacker group, who have been reported to specifically target organizations in the energy sector as well as companies in other ICS sectors such as industrial/machinery, manufacturing and pharmaceutical.
In my 4SICS talk I disclosed a previously unpublished comprehensive view of ICS software that has been trojanized with the Havex backdoor, complete with screenshots, version numbers and checksums.
Dale Petersen, founder of Digital Bond, expressed the following request regarding the lack of public information about the software trojanized with Havex:
If the names of the vendors that unwittingly spread Havex were made public, the wide coverage would likely reach most of the affected asset owners.
Following Dale's request we decided to publish the information presented at 4SICS also in this blog post, in order to reach as many affected asset owners as possible. The information published here is based on our own sandbox executions of Havex malware samples, which we have obtained via CodeAndSec and malwr.com. In addition to what I presented at 4SICS, this blog post also includes new findings published by Joel "scadahacker" Langill in version 2.0 of his Dragonfly white paper, which was released just a couple of hours after my talk.
In Symantec's blog post about Havex they write:
Three different ICS equipment providers were targeted and malware was inserted into the software bundles
Trojanized MESA Imaging driver
The first vendor known to have their software trojanized by the Dragonfly group was the Swiss company MESA Imaging, who manufacture industrial grade cameras for range measurements.
Image: Screenshot of trojanized MESA Imaging driver installer from our sandbox execution
|Product:||Swiss Ranger version 18.104.22.1686 (libMesaSR)|
|Exposure:||Six weeks in June and July 2013 (source: Symantec)|
eWON / Talk2M
The second vendor to have their software trojanized was the Belgian company eWON, who provide a remote maintenance service for industrial control systems called “Talk2M”.
Back in January 2014, the eWON commercial web site www.ewon.biz had been compromised. A corrupted eCatcherSetup.exe file had been uploaded into the CMS (Content Management System) of www.ewon.biz web site. eCatcher download hyperlinks were rerouted to this corrupted file. The corrupted eCatcherSetup.exe contained a malware which could, under restricted conditions, compromise the Talk2M login of the infected user.
Image: Screenshot of trojanized Talk2M eCatcher installer from our sandbox execution
|Product:||Talk2M eCatcher version 22.214.171.12473|
|Exposure:||Ten days in January 2014, 250 copies downloaded (source: Symantec)|
Prior to version 2.0 of Joel's Dragonfly report, eCatcher was the only product from eWON known to be infected with the Havex backdoor. However, Joel's report also listed a product called “eGrabit”, which we managed to obtain a malware sample for via malwr.com.
Image: Screenshot of trojanized eGrabIt installer from our sandbox execution
|Product:||eGrabIt 126.96.36.199 (version 3.0 Build 82)|
|Backdoor:||Havex RAT 038|
MB Connect Line
The most recent company known to have their software infected with the Havex backdoor was the German company MB Connect Line GmbH, who are known for their industrial router mbNET and VPN service mbCONNECT24.
MB Connect Line published a report about the Dragonfly intrusion in September 2014, where they write:
On 16th of April 2014 our website www.mbconnectline.com has been attacked by hackers. The files mbCHECK (Europe), VCOM_LAN2 and mbCONFTOOL have been replaced with infected files. These files were available from 16th of April 2014 to 23th of April 2014 for download from our website. All of these files were infected with the known Trojan Virus Havex Rat.
Image: Screenshot of trojanized mbCONFTOOL installer from our sandbox execution
|Company:||MB Connect Line GmbH|
|Product:||mbCONFTOOL V 1.0.1|
|Exposure:||April 16 to April 23, 2014 (source: MB Connect Line)|
|Backdoor:||Havex RAT 043|
Image: Screenshot of trojanized mbCHECK application from our sandbox execution
|Company:||MB Connect Line GmbH|
|Product:||mbCHECK (EUROPE) V 1.1.1|
|Exposure:||April 16 to April 23, 2014 (source: MB Connect Line)|
|Backdoor:||Havex RAT 043|
Notice how only mbCHECK for users in Europe was trojanized, there has been no report of the USA/CAN version of mbCHECK being infected with Havex.
We have not been able to get hold of a malware sample for the trojanized version of VCOM_LAN2. The screenshot below is therefore from a clean version of this software.
Image: Screenshot VCOM_LAN2 installer
|Company:||MB Connect Line GmbH|
|Exposure:||April 16 to April 23, 2014 (source: MB Connect Line)|
Conclusions on Havex Trojans
The vendors who have gotten their software trojanized by Dragonfly are all European ICS companies (Switzerland, Belgium and Germany). Additionally, only the mbCHECK version for users in Europe was infected with Havex, but not the one for US / Canada. These facts indicate that the Dragonfly / Energetic Bear threat actor seems to primarily target ICS companies in Europe.
Next: Detecting Havex with NSM
Read our follow-up blog post Observing the Havex RAT, which shows how to detect and analyze network traffic from ICS networks infected with Havex.
Posted by Erik Hjelmvik on Monday, 27 October 2014 11:11:00 (UTC/GMT)
A great way to enable digital forensics of control system networks is to implement network security monitoring. Captured network traffic is a great source for evidence when analyzing an attackers steps as he attempts to hack a SCADA system. The newly added support for the IEC-104 protocol in NetworkMiner also allows investigators and incident responders to see what commands the attacker sent to the control system.
We at Netresec recently announced the release of NetworkMiner 1.4, which comes with a parser for the SCADA protocol IEC 60870-5-104 (aka IEC-104). Bringing this Industrial Control System (ICS) protocol into NetworkMiner is a first step to support forensics of compromised ICS networks. The traffic from ICS networks does, of course, need to be captured (sniffed) in order to support network forensics; we are strong supporters of such network monitoring for ICS networks (read our “Monitor those Control System Networks” blog post for more details).
Why monitor ICS networks?
Computer forensics typically involves performing forensic analysis of hard disks. Disk forensics is very effective when analyzing a hard drive from a PC (like an operator workstation), but far more complicated when it is an embedded device like a PLC or RTU that is to be analyzed.
“My gut tells me that there is greater targeting and wider compromise than we know about. Why? Again, my instincts tell me that there is a lack of cyber forensics and response procedures at most of these facilities. If you do not have cyber forensic capabilities, it is hard to know if you have a cyber intrusion.”
Even though the hack was later shown to just be just a false alarm, David’s point about lacking capabilities for digital forensics and incident response for this type of critical infrastructure still holds true.
“We don't know how many other SCADA systems have been compromised because they don't really have cyber forensics.”
As Joe and David say, the ability to perform digital forensics in SCADA systems is truly lacking today. Our propose with this blog post is to inform control system operators that forensic data/evidence can be easily collected from ICS / SCADA systems by implementing a simple solution for network monitoring with full packet capture.
How to monitor ICS networks
The SCADA network diagram below has been sectioned into multiple security zones according to the zoning principle published by Jens Z, Iiro and me at CIRED 2009 (our zones align nicely with ISA-99 security Levels by the way).
The purple octagons represent interconnections between zones. Each such interconnection should be secured with perimeter protection, typically by a firewall, but we additionally argue that all network traffic passing through should be captured and stored as pcap files. Storing all network traffic this way makes it possible to perform network forensics on the network traffic after an intrusion is believed to have taken place.
We recommend a very simple setup, where a network tap is used to provide a copy of all traffic to a sniffer. An acceptable alternative to buying a network tap is to configure a monitor / SPAN port on a switch (see our sniffing tutorial “Intercepting Network Traffic” for more details on how to choose sniffing hardware).
Our recommended solution for the sniffer is to install FreeBSD with dumpcap (part of the net/tshark ports package). An even easier solution is to install Doug Burks’ Security Onion, which is a Linux distro built especially for network security monitoring. More about configuring a sniffer can be found in our second sniffing tutorial titled “Dumping Network Traffic to Disk”.
Analyzing captured IEC 104 traffic
Let’s assume the file 090813_diverse.pcap from pcapr contains network traffic from a suspected security breach at a hydro-power plant. Let’s also assume that parameter 4821 (i.e. IOA 4821 in IEC-104 language) controls the floodgates of the plant’s dam, where setting a value greater than 0% for this parameter would mean opening the floodgates.
By loading the pcap file into NetworkMiner and selecting the “parameters” tab we can see a nice log of all IEC-104 communication.
NOTE: We’ve hidden several fields (like IP, port, time etc) in the screenshot above in order to make it fit.
The following timeline can be extracted from the list of events provided by NetworkMiner:
- Frame 154 - The attacker sends command to set IOA 4821 to 50.354%
- Frame 156 - The RTU confirms the request
- Frame 162 - The RTU reports that the requested command has been successfully completed, i.e. floodgates are now open!
More ICS protocols
Would you like to see more ICS protocols in NetworkMiner? We’d be happy to implement protocols like DNP3, MODBUS, ICCP, Siemens S7, IEC 61850, etc. if you can provide us with captured network traffic! Please send an email to info[at]netresec.com if you are interested!
Posted by Erik Hjelmvik on Thursday, 30 August 2012 12:03:00 (UTC/GMT)
The Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) issued a security advisory for Siemens' SIMATIC Step 7 PLCs a couple of weeks ago. I've previously recommended asset owners to monitor the network traffic in their Industrial Control Systems (ICS), and ICS-CERT have followed my line of thinking by suggesting the following defensive measures:
"Configure an intrusion detection system (IDS) to monitor traffic for unusual or unauthorized activity.
- Monitor traffic on the ISO-TSAP protocol, Port 102/TCP.
- Monitor traffic being unexpectedly sent outside the automation network.
- Monitor traffic between workstations. This traffic may be indicative of attacker pivoting through your network"
The German ICS security cowboy Ralph Langner has written a somewhat confused blog post where he is critisizing ICS-CERT's advisory. In this blog post Langner says the following about ICS-CERT's recommendation to monitor the ISO-TSAP traffic:
"It would be interesting to learn how the authors of the advisory suggest this should actually be done. We wonder if they have ever peeked into the data traffic of a Siemens PLC’s port 102 in a real installation [...] In order to make any sense out of TCP port 102 traffic it is required to do deep packet inspection. Unfortunately, the details of the layer seven protocol that needs to be analyzed, along with certain peculiarities at layer four such as pre-defined binary TSAPs, are not documented by the vendor. So in essence what ICS-CERT suggests is that asset owners start reverse analyzing the S7 protocol in order to configure their intrusion detection systems, which seems like a far stretch."
So, is Langner saying that the Siemens S7 protocol is too complicated to be reverse engineered? If encrypted and strongly obfuscated protocols like Skype can be reversed, then the S7 protocol should be a piece of cake. I've manually reverse engineered multiple protocols when building protocol parsers for NetworkMiner, and I can testify that most unencrypted and non-obfuscated protocols can be reversed in just a few hours. It would therefore be quite simple for IDS vendors to add support for the S7 protocol to their software. I also believe that even a very rudimentary IDS functionality, which just checks which IP addresses that are communicating over TCP port 102, would provide value. Such a simple feature doesn't even require the IDS vendor to implement a parser for the S7 protocol or even the ISO-TSAP protocol.
Ralph also criticizes ICS-CERT's recommendation to "Monitor traffic being unexpectedly sent outside the automation network" by saying:
"While the advice per se might not be completely wrong, we don’t see any relation to the Beresford vulns which highlight the risk of process manipulation, not the risk of industrial espionage and exfiltration of trade secrets."
A machine on an ICS network trying to contact an external IP address is typical Indicator of Compromise, but Langner fails to understand this basic principle of network security monitoring and incident response. Malware very often use outbound connections to access Command-and-Control servers as well as to download additional software to maintain its foothold on the infected machine. I'm certain that this is why ICS-CERT recommend asset owners to monitor for outgoing traffic, especially since ICS systems normally don't communicate with external systems and typically don't host any confidential data or "trade secrets".
A point that ICS-CERT failed to stress, however, is the need for asset owners to also store the full content network traffic (pcap files) from their network monitoring installations. This is an absolute necessity when investigating an alert from an IDS in order to better determine if an event is a security incident or just a false positive.
More on capturing network traffic can be read in my blog post Sniffing Tutorial part 2 - Dumping Network Traffic to Disk.
Posted by Erik Hjelmvik on Wednesday, 24 August 2011 14:47:00 (UTC/GMT)
Network security monitoring is an ideal security feature to apply to industrial control system networks. Owners of the IT-systems that control our critical infrastructure have unfortunately not yet understood the usefulness of monitoring their own network traffic.
SCADA security has in the past few years become a hot topic at mainstream hacker conferences like BlackHat and DEFCON. Stuxnet has also increased the interest for SCADA security even more in the “traditional” IT security and hacking community. This interest has caused security researchers to find and publicly disclose multiple vulnerabilities in SCADA and Industrial Control Systems (ICS).
Having worked with IT security for a major electric utility company (in the pre-Stuxnet era) I know from my own experience that the resilience against network based hacking attacks varies greatly between different brands and models of PLCs and RTUs. But an attacker with access to a control system network don't need to use any vulnerability to control a PLC. The reason for this is that the communication protocols used by these embedded devices don't use authentication. The attacker can therefore simply send any commands he wishes to the PLCs to make them open a dam gate, blow a generator or spin a centrifuge out of control; no vulnerabilities needed!
This morning I read a blog post titled “PLC’s: Insecure By Design v. Vulnerabilities” written by Dale G Peterson (an old friend from my SCADA security days). In this blog post Dale stresses the fact that many control system devices are “Insecure By Design”. He also mentions Secure DNP3 (an encrypted SCADA protocol that basically is an American fork from the IEC 60870-5 standard), which can increase security by introducing authentication and encryption.
I've always been against introducing any form of encryption in control system environments since availability is what's needed in these environments, not confidentiality! Adding encryption is also against the KISS principle, which should always be a foundation when designing control systems.
Instead I see two major efforts that the ICS community need to carry out in order to achieve better security. The first effort is to segment the ICS networks into different security zones and apply appropriate perimeter protection between the zones. The second effort is to establish proper network security monitoring of the control system networks.
Segmentation and perimeter protection are nowadays widely accepted measures in the ICS community. There are even special ICS firewall vendors, such as Tofino, RuggedCom and Moxa. Even crazy concepts such as “unidirectional gateways” are successfully used to protect critical ICS networks.
But the concept of network security monitoring has, on the other hand, not really been grasped by the ICS community yet. It actually seems as if they don't even understand the value provided by monitoring the network traffic in control systems. The ANSI/ISA-TR99 standard does, for example, mention “sniffing” several times, but only in the context of sniffing being a threat rather than treating it as a security control. The DHS document “Cyber Security Procurement Language for Control Systems” even contains this somewhat absurd statement:
“Scanning is an effective tool to identify vulnerabilities. Use caution, however, because active scanning of live control system networks has been known to disable the networks during operations. FAT and SAT provide critical opportunities for active scanning tests without an impact to production. Even passive scanning is not recommended on production systems until the impact to operations is fully understood.”Ignoring the fact that they write “passive scanning” when they refer to “sniffing” or “network monitoring” I can't really believe that DHS recommend control system owners to avoid monitoring their own network traffic. Shame on you DHS!
I therefore take it upon myself to educate the authors of these misguiding standards as well as control system owners as to why they should monitor their networks. Here are six good reasons for why process control system networks should be monitored:
- Embedded devices used in control systems do often have poor or none-existent host based security logging. Event logs as well as security logs can be built simply by sniffing and analyzing sniffed network traffic, without having to introduce any additional complexity to the embedded devices.
- There is usually no centralized administration of the devices on a control system network, and network diagrams often differ significantly from the reality. Performing an NMAP scan of a control system network isn't suitable since that actually can cause some devices to crash (trust me!), but an inventory of the devices on a network can easily be created simply by sniffing network traffic. See my article “Passive Network Security Analysis with NetworkMiner” for more details.
- Viruses and worms can get to even isolated/air gapped networks, either through USB flash drives or through infected laptops that get connected to the isolated network. Many viruses can be detected simply by looking at the network traffic they generate when they attempt to establish a connectionto a command-and-control server. In the case with Stuxnet, for example, the infected machines would try to establish connections to the domains mypremierfutbol [dot] com and todaysfutbol [dot] com. Any attempts to lookup external DNS names from within an isolated network are always worth looking closer at!
- Prevention eventually fails, i.e. no matter how secure you think your network is someone or something will eventually penetrate your perimeter protection. So you'll better be prepared!
- Assume your network security perimeter has already been breached.
In a recent report from McAfee Dmitri Alperovitch (VP Threat Research at McAfee) writes:
“Having investigated intrusions such as Operation Aurora and Night Dragon (systemic long-term compromise of Western oil and gas industry), as well as numerous others that have not been disclosed publicly, I am convinced that every company in every conceivable industry with significant size and valuable intellectual property and trade secrets has been compromised (or will be shortly), with the great majority of the victims rarely discovering the intrusion or its impact. In fact, I divide the entire set of Fortune Global 2000 firms into two categories: those that know they’ve been compromised and those that don’t yet know.”The best way to find out if you are infected is to monitor your network for suspicious traffic.
- Network security monitoring is simple and doesn't affect the network being monitored! Check out my sniffing tutorials “Intercepting Network Traffic” and “Dumping Network Traffic to Disk” to get an introduction.
Now, go out there and sniff those process control networks! ;)
Posted by Erik Hjelmvik on Wednesday, 03 August 2011 18:56:00 (UTC/GMT)