Unmasking Malware Families
Identifying malware families is hard. Malware samples are often packed, strings encrypted and configurations may only appear after several stages of execution. Even experienced reverse engineers can get the malware family wrong and commercial sandboxes do not always produce correct classifications.
Malware C2 network traffic can provide another way to approach the problem. C2 traffic often contains enough information to identify the application-layer protocol and the malware family using it.
FlowCarp
FlowCarp identifies application-layer protocols in network traffic by analyzing observable characteristics of the traffic. It computes statistical measurements from a capture and compares them with models for known protocols.
Unlike traditional IDS signatures, FlowCarp protocol models do not describe protocols using specific strings or byte sequences. To add detection for a new protocol to FlowCarp, you provide example traffic that is used to build a statistical model.
This makes it possible to identify proprietary protocols, including C2 protocols used by malware.
FlowCarp can detect hundreds of malware protocols, so I often run it against PCAP files captured during sandbox executions. It often correctly identifies which C2 protocol a sample is using and points out the IP and port number of the C2 server.
Reporting Misclassifications
I occasionally contact cybersecurity companies or malware researchers when FlowCarp indicates that there's a mismatch between a sample’s reported family and its C2 protocol.
Some decide not to update their publications, while others revisit their findings after being informed of the misclassification. Companies often don’t provide public credit for such feedback. Instead, they typically revise their publication with updated findings and thank the broader research community, with wording like: "We extend our sincere gratitude to the researchers who provided critical feedback" or "Thank you to the community members who prompted us to take another look".
Brad Duncan is a notable exception. His malware-traffic-analysis.net writeups regularly give credit after receiving useful corrections or additional analysis.
Statistical Classification and Signatures
FlowCarp takes a different approach from traditional signature-based network detection.
A typical IDS signature might look for a known byte sequence, string, header, protocol field or other recognizable pattern. This works well when the pattern is stable and distinctive. It is also easy to understand: the rule matched, so the alert was generated.
FlowCarp's statistical classification looks at several traffic characteristics, such as:
- Packet sizes
- Timing between packets
- Message length distributions
- Request and response behavior
- Protocol structure
This approach can be useful when recognizable strings or byte sequences are unavailable. Encryption may hide the contents of a message, but it does not necessarily hide every recognizable property of the network protocol. FlowCarp may still be able to distinguish traffic based on its structure and behavior.
Statistical protocol detection doesn't require hand-built rules or deep protocol knowledge. Collect some example traffic, build a model, and FlowCarp automatically identifies the distinguishing characteristics.
Reducing Alert Overlap
A single network connection might generate IDS alerts associated with multiple malware families. This can happen when a generic signature matches traffic from several different families or when multiple signatures happen to match the same traffic. An analyst may therefore receive several possible family labels for one connection.
FlowCarp takes a simpler approach: it assigns a single protocol or malware label to each flow.
Traditional IDS tools are still great for spotting malicious traffic and identifying infected hosts and C2 servers. FlowCarp is not intended to replace those tools. It works best as an additional classification layer. It can help identify protocols that do not have useful signatures, provide a more specific label for some traffic and simplify the process of adding detection for new protocols.
Adding a New Protocol
Suppose you have identified a new C2 protocol used by a malware you call MalwareXYZ.
You collect several C2 sessions from sandbox executions and save them in a capture file called xyz.pcap. If possible, use sessions from more than one execution or sample rather than relying on a single session.
You can then use those examples to build a custom protocol model:
The model is given the label MalwareXYZ, marked as MALICIOUS, and saved as malwarexyz.fcp. Once the model has been created, you can use it to classify traffic in another PCAP:
This lets you test the model against captures from other samples or sandbox executions.
The quality of the result depends heavily on the examples used to build the model. If the training data contains only one narrow variation of the protocol, the model may not generalize well.
Trying the Demo Server
There is a free FlowCarp demo server at demo.flowcarp.com. It runs a FlowCarp web service with a commercial license and can identify more than 600 protocols.
The demo server provides a quick way to evaluate FlowCarp's protocol identification feature before building a local workflow. To submit a PCAP, use:
The server returns a JSON response containing detected flows, protocols, and alerts. For example, the output can be passed to tools such as `jq` or processed by an existing incident response pipeline.
C2 Traffic as Additional Evidence
FlowCarp does not replace reverse engineering, IDS signatures or analyst judgment. Its value is that it gives analysts another way to examine a sample. Statistical protocol classification can be a practical addition to malware analysis and incident response workflows. When a sample’s code and configuration are difficult to analyze, its C2 traffic may still reveal the underlying malware family.
Posted by Erik Hjelmvik on Monday, 21 September 2026 10:00:00 (UTC/GMT)
